{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dyna Q\n",
    "\n",
    "We will be using **Dyna Q** on Cliff World environment as given below:  \n",
    "\n",
    "![GridWorld](./images/cliffworld.png \"Cliff World\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q Learning Update equation\n",
    "\n",
    "Q Learning control is carried by sampling step by step and updating Q values at each step. We use ε-greedy policy to explore and generate samples. However, the policy learnt is a deterministic greedy policy with no exploration. The Update equation is given below:\n",
    "\n",
    "$$ \n",
    "\\DeclareMathOperator*{\\max}{max} Q(S,A) \\leftarrow Q(S,A) + \\alpha * [ R + \\gamma * \\max_{A'} Q(S’,A’) – Q(S,A)] $$\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initial imports and enviroment setup\n",
    "import numpy as np\n",
    "import sys\n",
    "import random\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import gym\n",
    "import gym.envs.toy_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Q- Learning agent class\n",
    "from collections import defaultdict\n",
    "\n",
    "class DynaQAgent:\n",
    "    def __init__(self, alpha, epsilon, gamma, get_possible_actions, n):\n",
    "        self.get_possible_actions = get_possible_actions\n",
    "        self.alpha = alpha\n",
    "        self.epsilon = epsilon\n",
    "        self.gamma = gamma\n",
    "        self.n = n\n",
    "        self._Q = defaultdict(lambda: defaultdict(lambda: 0))\n",
    "        self.buffer = {}\n",
    "\n",
    "    def get_Q(self, state, action):\n",
    "        return self._Q[state][action]\n",
    "        \n",
    "    def set_Q(self, state, action, value):\n",
    "        self._Q[state][action] = value\n",
    "\n",
    "    # Q learning update step    \n",
    "    def update(self, state, action, reward, next_state, done):\n",
    "        if not done:\n",
    "            best_next_action = self.max_action(next_state)\n",
    "            td_error = reward + self.gamma * self.get_Q(next_state, best_next_action) - self.get_Q(state,action)\n",
    "        else:\n",
    "            td_error = reward - self.get_Q(state,action)\n",
    "            \n",
    "        new_value = self.get_Q(state,action) + self.alpha * td_error        \n",
    "        self.set_Q(state, action, new_value)\n",
    "            \n",
    "    # get best A for Q(S,A) which maximizes the Q(S,a) for actions in state S\n",
    "    def max_action(self, state):\n",
    "        actions = self.get_possible_actions(state)\n",
    "        best_action = []\n",
    "        best_q_value = float(\"-inf\")\n",
    "        \n",
    "        for action in actions:\n",
    "            q_s_a = self.get_Q(state, action) \n",
    "            if q_s_a > best_q_value:\n",
    "                best_action = [action]\n",
    "                best_q_value = q_s_a\n",
    "            elif  q_s_a == best_q_value:\n",
    "                best_action.append(action)\n",
    "        return np.random.choice(np.array(best_action))\n",
    "        \n",
    "    # choose action as per ε-greedy policy for exploration\n",
    "    def get_action(self, state):        \n",
    "        actions = self.get_possible_actions(state)\n",
    "\n",
    "        if len(actions) == 0:\n",
    "            return None\n",
    "\n",
    "        if np.random.random() < self.epsilon:\n",
    "            a = np.random.choice(actions)\n",
    "            return a\n",
    "        else:\n",
    "            a = self.max_action(state)\n",
    "            return a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot rewards\n",
    "def plot_rewards(env_name, rewards, label):\n",
    "    plt.title(\"env={}, Mean reward = {:.1f}\".format(env_name,np.mean(rewards[-20:])))\n",
    "    plt.plot(rewards, label=label)\n",
    "    plt.grid()\n",
    "    plt.legend()\n",
    "    plt.ylim(-300, 0)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#training algorithm\n",
    "def train_agent(env, agent, episode_cnt = 10000, tmax=10000, anneal_eps=True):\n",
    "    episode_rewards = []\n",
    "    for i in range(episode_cnt):\n",
    "        G = 0\n",
    "        state = env.reset()\n",
    "        for t in range(tmax):\n",
    "            action = agent.get_action(state)\n",
    "            next_state, reward, done, _ = env.step(action)\n",
    "            agent.update(state, action, reward, next_state, done)\n",
    "            G += reward \n",
    "            if done:\n",
    "                episode_rewards.append(G)\n",
    "                # to reduce the exploration probability epsilon over the \n",
    "                # training period. \n",
    "                if anneal_eps:\n",
    "                    agent.epsilon = agent.epsilon * 0.99\n",
    "                break\n",
    "            # add the experience to agent’s buffer (i.e. agent’s model estimate)\n",
    "            agent.buffer[(state,action)] = (next_state, reward, done)\n",
    "            state = next_state\n",
    "            # plan n steps through simulated experience\n",
    "            for j in range(agent.n):\n",
    "                state_v, action_v = random.choice(list(agent.buffer))\n",
    "                next_state_v, reward_v, done_v = agent.buffer[(state_v,action_v)]\n",
    "                agent.update(state_v, action_v, reward_v, next_state_v, done_v)\n",
    "                \n",
    "    return np.array(episode_rewards)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# helper fucntion to print policy for Cliff world\n",
    "def print_policy(env, agent):\n",
    "    nR, nC = env._cliff.shape\n",
    "\n",
    "    actions = '^>v<'\n",
    "\n",
    "    for y in range(nR):\n",
    "        for x in range(nC):\n",
    "            if env._cliff[y, x]:\n",
    "                print(\" C \", end='')\n",
    "            elif (y * nC + x) == env.start_state_index:\n",
    "                print(\" X \", end='')\n",
    "            elif (y * nC + x) == nR * nC - 1:\n",
    "                print(\" T \", end='')\n",
    "            else:\n",
    "                print(\" %s \" %\n",
    "                      actions[agent.max_action(y * nC + x)], end='')\n",
    "        print()\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "    This is a simple implementation of the Gridworld Cliff\n",
      "    reinforcement learning task.\n",
      "\n",
      "    Adapted from Example 6.6 (page 106) from Reinforcement Learning: An Introduction\n",
      "    by Sutton and Barto:\n",
      "    http://incompleteideas.net/book/bookdraft2018jan1.pdf\n",
      "\n",
      "    With inspiration from:\n",
      "    https://github.com/dennybritz/reinforcement-learning/blob/master/lib/envs/cliff_walking.py\n",
      "\n",
      "    The board is a 4x12 matrix, with (using Numpy matrix indexing):\n",
      "        [3, 0] as the start at bottom-left\n",
      "        [3, 11] as the goal at bottom-right\n",
      "        [3, 1..10] as the cliff at bottom-center\n",
      "\n",
      "    Each time step incurs -1 reward, and stepping into the cliff incurs -100 reward\n",
      "    and a reset to the start. An episode terminates when the agent reaches the goal.\n",
      "    \n"
     ]
    }
   ],
   "source": [
    "# create cliff world environment\n",
    "env = gym.envs.toy_text.CliffWalkingEnv()\n",
    "print(env.__doc__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a Dyna-Q Learning agent\n",
    "agent = DynaQAgent(alpha=0.25, epsilon=0.2, gamma=0.99, get_possible_actions=lambda s : range(env.nA), n=20)\n",
    "\n",
    "#train agent and get rewards for episodes\n",
    "rewards = train_agent(env, agent, episode_cnt = 500)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAEICAYAAAC3Y/QeAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAABGYklEQVR4nO2deZgc1XXof2cWSaMd7buQkNgkhEASCGRgwCxisZET7BAvLImfwPH6yHMCJgsE87zG8cOJIRgDAdsheMHYYIGRTYtNgCVAQkIL2jWSQBuSZrTOTN/3R1X1VHffqq5eZ6br/L6vv66699ate2s5dercc0+JMQZFURQlXtR0dgMURVGUyqPCX1EUJYao8FcURYkhKvwVRVFiiAp/RVGUGKLCX1EUJYao8FdCEZEbROQl33qLiEx0lxtE5Lcisl9Efu6mfV1EdovIe53Q1k0icnFAXqOINFW6TdWOiBwvIkZE6jq7LUp+qPBXEJHLROQFEWkWkV0iskhEPmora4zpa4zZ4K5eAwwHBhtjPi4iY4G/BU41xoyw7GeNiHzCtz7HFRyZaS2dKUzcB4URkV9lpJ/upic6qWmxRUQuFJHnXUVjkyX/effaPSAiy0Tk6pC6RES+JSJ73N+3RUTK2oEuiAr/mCMi1wA/Bx4BxuAI838CPhJh8/HAWmNMm299jzFmZ0D5F4ALfOvnA6staa/46ozSh3I8KHYB54rIYF/a9cDaMuwrb0SkthP22Zna/UHgQeCrAflfBkYaY/oD84GfiMjIgLLzgXnA6cA04CrgppK2thugwr+TEZFRIvJLV2vZKCJf8uXdISKPi8gjrla+UkRmunm3isgvMur6fyJyTx77FuB7wF3GmAeMMfuNMUljzCJjzP8K2MaIyCQRuRPnIfEXrqZ+E/AcMMpdf9iy+Qs4wt3jPOBblrQX3H191O3zPhFJiMgpvnZsEpG/F5HlwMFMweSapB4WkQ9E5B1gVtTj4nIM+DVwrVtfLfAJ4KcZ+zlZRJ4Tkb2WN5srReRNVxvdKiJ3+PI8c8n1IrLFNZXdHtQYty/3isjvROQgcGHQtSMivUTksIgMcdf/QUTaRKS/u/51Efl+Hm38axHZAvxRRGpF5LtuezcAV+Z5XAvCGPO6MeZRYENA/nKfwmCAemBsQHXXA/9qjGkyxmwD/hW4ocRN7voYY/TXST+ch+9SHCHaA5iIc3Ff5ubfARwBrgBqgW8Ar7p544FDQH93vRbYAcx2138I7Av4LXfLnIxzo0wIaeMNwEu+dQNM8rXvJ768RqAppK5xQBIY5PZ9J9AAbPWl7cN5GJyIo+1dgnMj/x2wDujh1rUJeAvnBm/wpV3sLn8TeNGtdyywIqxtGe1sBJqAc4HX3LQrgGeBzwIJN62P2/YbgTrgTGA3MMVXz2luv6YB7wPz3Lzj3WP5I/cYnA4cBU4JaNPDwH5gjltfb8KvnReAP3eXfw+sBy735X0sjzY+4va1AbgZ521trHtsn3fL1AW0+ymCr8OnCrhnLgY2hezriNueZ4CagHL7gbN96zOB5s6WB5X+qebfucwChhpj/sUYc8w4tvQf4WqbLi8ZY35njGkHHsUREhhjNgNv4Ly+AlwEHDLGvOrm/40xZmDAb5q7jWfS2FHWXroYY7YAW3C0+9OBd40xh4GXfWm9gNeAvwCeNsY8Z4xpBb6LI3zO9VV5jzFmq1tHJp8A7jbG7DXGbAUivxH52vsKMEhETgKuwxGCfq7CEUQPGWPajDFvAL/EGQvBGJMwxrxtnLep5cB/k27iArjTGHPYGLMMWOYegyCeNMa8bIxJ4gjssGtnEXCB+0Y0ze3/BSLSC+e6ezGPNt5hjDnoHudPAN93j/teHIUk7BheFXIdXhW2bb649fXDfVC7x8lGX5wHgMd+oG/c7P4q/DuX8Thmkn3eD/gajt3dw+81cwjo5TNx/Az4S3f5k+56Puxx/4Nso+XAM/2cjyuAgJd8aa8ZY44Co4DN3kbujbwVGO2ra2vIfkZl5G8OKpiDR4EvABcCT2TkjQfOzjh/nwJGAIjI2b6ByP04WvOQjDoyz2/fkLb4+5Pr2lmEo9WfCbyNY5K7AJgNrDPG7M6jjf79luq4BiIiX3NNhy0icl8+2xpjWo0xC4DLJMBpAWgB+vvW+wMtxn0NiAsq/DuXrcDGDG2onzHmiojb/xxoFJExwMfwCX8Ruc93A2X+VrrF1rht+PNSdioHnvA/jw7h/6Iv7QU3bTuOgANS4xNjgW2+usJu1h2k23zHFdjeR4G/AX5njDmUkbcVWJRx/voaYz7n5v8M+A0w1hgzALgPKEa79Pc317XzCnASznWxyBjzDs4xuBLnweARpY3+/eZ1XEVkQch1uMDaSWP+r3sc+xpjbg6rP4Q64ISAvJWkv2Gd7qbFChX+ncvrwAF34LLBHUybKiKRBieNMbuABPAQjiBY5cu72XcDZf6muGUMcAvwjyJyo4j0F5EaEfmQiNxf8t46vACcgaOFvuymvQ1MwNGuPeH/OHCliHxYROpxXEiP4gi1KDwO3CYix7kPxy/6M90B1IdzVWKM2ei21TYY+xRwooh8RkTq3d8s38B0P2CvMeaIiJyF83ZWKkKvHfdBtRT4PB3C/hUcrxa/8M+3jY8DXxKRMSJyHHBrWGFjzOUh1+HlUTvrXpe9cMZ/xB3U7uHmnSwil7vHoV5EPo2jTCwKqO4R4BYRGS0io3CurYejtqVaUOHfibh2/I8A04GNOIOFDwAD8qjmZziDYPmafLw2/ALHvv5XONr2+8DXgScLqS/C/tbiDPTuMMbsc9OSOMKsP65wN8asAT4N/ADnuHwE+Igx5ljEXd2JY5LYiDPg+WhG/lg6Hj652vySMWa7Jb0ZuBTHzr4dx4TzLaCnW+RvgH8RkWacgdnHI7Y9SpuiXDuLcITl6771fnQ8YAtp449wBr6X4Yw5/Sq8eMk4HzgM/A7nbeMwznkF503lDpzraheO2+dfuGMwiMh5ItLiq+s/gd/iKB0rgKfdtFghMTNzKQquxrgMmOYOJitK7FDhryiKEkM6zewjInPFmRSzTkRC7YaKoihKaekUzV+c2ZJrcSbwNAF/Av7S9UhQFEVRykxnaf5n4fgab3AH8B4DAgMxKYqiKKWlswI1jSZ9okgTcHZmIRGZjxOEiYaGhhljxwaF6ggnmUxSUxMvxybtczzQPseDYvq8du3a3caYoZnpnSX8bRNdsuxPxpj7gfsBZs6caZYsWVLQzhKJBI2NjQVt213RPscD7XM8KKbPImKdhd1Zj88m0mcJjsHxk1YURVEqQGcJ/z8Bk0VkgutzfS3OFHNFURSlAnSK2ccY0yYiX8CZKVgLPGiMiV1sDUVRlM6i077MY4z5Hc5UbUVRFKXCxGvIXFEURQFU+CuKosQSFf6KoigxpNNs/l2Jo23tnHX3H9h/uJUvXTSJWy49iV3NRznnG3/gv+fPZsHb77Fl70EeuN4Js//Aixv47fId/M/82by2cS9/85OltCUNR9ucr8ZdedpIrpkxhr//5XL+pvEEvvnManr3qOOLF03ip69t4bH5s/nMj19n1Y4D3DjneF5Yu4trZozlc40n8Jkfv8apI/uzcNX7rN91kAEN9Tx+0zl8c8Eq3tiyj/2HW/nHq07lrz80gQ//a4JrZ43juD49+D8/X0av+hrqamo41p50JoUsXMCwfj156osf4vk1O/nHX6/EGMP/vuREfvDHdZx7wmBEYOGqnXz45GHc++kZ/OiFDXz392sY0rcnD1w/k1NG9ufbz6zmgZc2UitCz/oa9h3qCIRZI5A00KOuQ4+oqxEOHWunR10NF5w4lL+99ERuePBP7G45Sk2NM8WjPWloqK9FBI62JakVoUagNelM9+hVV0N70qTWAY65x7ehvpakcY730H492dV81KnLtPPzE/fz6OLN/OrNbZw6sj/Txw7kZ69vSdXx0dNHcfBoG39YvTOt3ob6Wg63Om2urxFqa4SkIdU+P8fakvTpUZvWNj/+/l900jBE4A+rd3KsLUnvHrUcaW2nvrYm9Es0uTjWlqS2RhAMNQsXpKWPG9SbedNHcf+LG7jh3An8dtl2drUcBQPH2pNp56pS1Iqkjm9YmdoaoT1p+NtLT+T7C9/lWHv2lxi9a7tU9OlRy5HWJO1dOMjlv1/Yq+R1dpuonuWc5LWr+Siz7l4IOILl1a99mLeb9vPpH7/GmeMG8saWfQA8+5XzeeLNbdy3aD0A3/7zaRw40srXn14VVDXHD+7N+weOcri1PZV22ugBvL1tf1bZJf9wMTO/vjArfcb441i6+YO0tH+86lTuesoeCuncEwYzMHmAg/UDWbR2Fx85fRS/XeZMoxjWryc7m4+myo4c0Isd+4/QUF/LTRdM5PsL32Xi0D7sP9TKkL49+crFk3nsT1tZtHZXYB8BGk8ayskjnC/jecdnyqj+rNx+ICWgAa44bQQHj7YH1nfe5CHs2H+EdTud8Os3zjmennW1LNu6j8Ub9li38ehVC4P7NXC4tZ2DR9tSQnvUgF58dPponlmxg4YedTQfaaVPjzouPHkYr23cw5vu+QXoUVuTJXBOHtGPxpOGAbDmvQM8v8Zp+80X2D8U5fW/Z11Nqg3esfAY0b8X884Ybd0+F4eOtfHI4s3MnjiIQaaZceOcj2m9t/8wv37LPl3mxjnH89DLmwD49Oxx9O1ZX9C+C8U7Jh87YzTD+9sFmVfGY0jfHlwzI3tW/5YtW1J9Lpanlm+n6QPnE9BB57MrMLPnDi6+6MKCthWRpcaYmZnpsdL8k0mDCIR9p/lwazs/eXUzH5rkfMb07W37UwLysu+/kFb2tY17OWVkPwBuOn8i//nChqz6Nu05xFXTRrJo7S6aj7QBsGrHgaxyAMu27ktbnzysL1NG9bfe0EGCH+CSU4czofUI7/cZwaK1u9i61/n64N0fm8rGXQd54KWNqbI79h9J9fv7C98F4JoZY0is2cXrG/fyuZ++wdhBDdb9eH0eN6g33/346Qzp63zDZNyg3jz66mYeunEWH/uPV3j/wJHUNrfOPYX3m48ECv8rThvJaxv2pIT/LZecSL9e9Wzbd5hr7n0l1d5MfvipM1mw+G1+u8G5kS88aSgvrdtNa7th/OA+3Hr5yWzY1cKWvYcwBqaNGcCtl5/Mxt0H+fQDr3HtrLE8vnQrs44fxK/e2JZW9xnjBnLr5ScD8O77zTy/ZhcDe9en0jI5eUQ//v35dUwe1pcFK5xP9N77qRl87qdLGdBQzyvr96TVWQj/67yJjBzQi5defIHGRqeeY21J1u1qYcW2A/Sqr+Gm80/gRy9u4LzJQ/jnj0yhob6Wd3Yc4K6rp4beA+VgRP+e/OrNbXz7mmnU19q1//98YT1+XfSK00Zaj1Ei8V6qz8WyascBmj44TF2NFHU+yk0i8V7uQnkSK+F//UOvM7x/L7778Y7Pdx5/69NccGJH2IuJQ/rwnWfXpARWa7thhCv8/Ywd1IAxJnWxfuGiSTz08ibra+r0sQPZd6iVl9bt5m8vOZEvfngyAKvfO8Dc77+YKveH1TsRgZV3XkbvHs6pWbXjAL9+aztTR/dnxTb7Q+MTM8cwb/poPvnAawDUuqYV7wY/rnc9g/v04FNnj+e7z64JPUaXnjqcPz9zDC/4hPPWvYezyl18yjBuu+IUbrvilKy8T549jk+e7WhmL996EQAn3r6AY+1Jxg5qYEi/HgD061lH89G2tG0d80+HYPKWRw9sYPFtH04r+6Fv/ZGmDw4zZVR/rjhtJA27V/PbDY5JqkddDfW1NbS2t+OFRHFMOYakMal6Jwzpk2rjFz88mW8/szqrP35BOWlYX26+4AQunzoiq5zHvDNGM++M0Xz158tSaf0b6nj6S+dxpLWdrz/9Dp9rnBS4fRTGDuqdldajroanvnheWtr/vuTE1PLfze084XbDnAncMGdCaJlaEdp80r+hR225m5W6VzxzZJyIjfBfsW0/L767myF9e2KMQUTwTF6eFnrlaSP58sWTufTfXmDbBx0Cz2/a/eGnzmTcoN584WdvkDQG41puRYQgZWpAQz13fHQKi9fv5iOnj0ql12RssLv5KD1qa1KCH+CUkf156MZZ1NfU8Okfv2atf/zgPmkCylv26m9LmlRaLoXvOx8/nQEN9Vlt8/PVy07i07PHB+bb+MPfXsCeg8cQEXr3qOOx+bNpqK/l6v9I/5Ji5ptZWDtqMvqZuV1mfo04dvz2pCEoRpZtf365IBJdQ6yxnJNe9bV8fd5pkbaPG87xMhnr5d5n+n+ciI3wf2TxJgB2txxl+/4jjB7YkKV1nn/iEI4f3AcgbfDHGMN5k4fwH586k/69HFupJ0i8YkLwxVojwqRhfZk0rG9Genq5tqSx1nHhScNYYRkj8BjUp0daXd6y93bdnjSptFzXeMfNEFyy8aShDGjIz2Y8dlDvNG119sTB7Nif/UZR4w78eoTd/7Yb1xswrPE9jGt8Dz5H8w82/dmEQKFCyP+AiaNwyZfMw1yJY5apKMWJWLh6btrfnjKdANyXWM/Pl2xlT0v6t8D92nu7T91PGkN9bU1K8DtlcTX/jvWgizVIy8wUQG0+IZ1dNjvtrOMHAXDC0L5pr62ZGm/aQyXHRd5hMgouU6obpdamZdek1x9F80/X+HHT/Meho7wxzsM8+DjbNP/C+hv1DUZxyDxGldX843d+YiH8n9/aRo/aGr7m2qcffXUzX/3Fcv7oc/cD5wLwLoFkmuafLdg7BImzLkjgBRQ1vT2ZDCwrFp39s+dN4MnPz+GsCYOsmr8nfPLT/HNrQqW6UYIEbVSNuUOzz67T/wbhfwhk2vxt+4+SFoX0cxI/4ZIvmee6EoPS/rfCuBEL4d+WdOzuw/r1TEv/4+r3gQ4fdf/Fl675Z1+INeKYFzps/sGDRsGCJqOd7SbwIrSl19UKp48d6ObbbP5uvXnY/G0CNVe7C8VWj/P2VZjN319njfhvbL/N3zH7RD0nQWlRSLf5F1ZHnMjW/Cu3zzg+nGMh/A3ezZd+gl9e5/iND2zosON7giLpc9qxmQlSZh/fYHCg2Sey5m8CHyC2KoKEZOYF3Z5MprRp2xuErU1hN0OpNLKgwdXoNn+b8HfTaiTrISgiJJOEa/6W41+oJ0hU85XikG3zL/8xi6LsVCsxEf4mayDRjzd4KYLV7GMTFt6Ar4ffxpxJVDt+a8CAL9iFdm2acMneX0rzb++oN9f9ZLOjZ5cJryMqQSYWf1vD2mEbxqj1PRBsZh9jDMk8x1YKlUFiOSdKMJkPWdX8y0s8hL9JNwNkkvL19eW3m3SzT5bwr8H183fNPkigoArSHGsz0h2bv70PtqqDNMtMAd7ue6jkuqGiuL6VzOZvufr85ynXfmzl/Jpcx1tAR/l2z+wTdE5KavNXzT8fsu+x8h+zzPGxOBEb4S8SPCDb5qrwfvfAZIa3j+2V1G/2EQkWmFHNPo7NP0jzt23vy7cs1/iEv0S8yKO4vmU+tArFtg+/x1XOB1VN+r+znK35i+8hkDT28xnWpsJt/h3LMZQteWNzqij/PqMpRdVIPIQ/ngnBnt+eEv4dgsJv0jEWzV9cs0/K1ZPizT7teZsjOhL9Ajnzgg6aPxDWziD31KC2FEKQT30U0xOkm3j823vb1mS80XkT+2zn0yPXG1Y+2CZ5KcFkO1WUf5+Z10iciIXw98w2Qee3zR3d9V98mX7+2VoJGZp/8JhCXgO+gRdhuEYaNuDb1t5hTgq7xv0PkHCbf/k0f/+Ab66b3/aWYvP28ad5mn8+56pQwZ054KyE0zmav33fcSAWwt8QbvNva+/Q/D2iDPgaQ4erJyGzRiO6gLaG+fnbNNKabKHnX/a28Wv+Yd4++bpYFkuQlh3d5p/+79/GPuCbHdsnqM5caVHINL8p4djeriu1zzjG9omF8Id0oZKJ3+YPHW6cHslk9oVYa7P5B8aLiZbeHubnn2P7sNg+7RH9/G0PkFzlisGuZUefch/m6llbk+415OUlkzn8/G2ungWbfYrbPm50hp+/hneocpIm3CyTsvnXZKcBbiC49G28B0RHeIf8Z/jawztEK5uZZjd9SKre2pSffzBRvVPK6+fvH6iNtn1QeIdMzVvEP74T/TgXO8krhnKlIDKVJzX7lJdYCH9jXKEXcIJb29Nt/kL6gG8yILxD0oB/llc+JhunfPp66ICvdfsgs0+6rdmx+ecniKKODRRDrgHfXDe/WG7c9PAO6cehRsT3lhe9TcXb/GMoWQqgMzR/9fOvcgxkBQzzk6kNiki65o/F5u/5+ZN7lmBkV89Qzd+2vb2u1Gxen+bf8WDLrsgmzDPbURcwvlAMQVp2VG3MavZJ+fRn59f4Nf+I4zBBaVGI+gajOHSOzd/bV9l31eWIh/A30Wz+/gk+UWL7eCGdxZfWkd9RNkhTzvbzT4bY/MOFkn+7TC+TtMBulvrtwj+4TDlvSn9sn1z7sfn5+90/JdXnjjTvXAdVbZ/kFbX1mdvFUKIUQebhqsTxi/PbWTyEPxFt/p6wIDOqpy22j6Q+5mK7gOp8Eimq736+mr8/zb9dSgDWdGj+YRd3XQTN3//pvXK+jvs19ly7sdv8O9KyzD6WuRCZpD9Ew8vmIo525GJQs09liYfwd232QZpke0ob9ARHpqun/cLM1Pz9Repqc2vK2eEdok3G6miDXZhlCj3/sq0tqRj+vrQs76YIgrMU1NQEP6QzsT10/SY4f1gHLy21n4B9pNWVUWe+6MSu/MgaV6toeIey76rLEQ/hT/qXnYLwX2v+T/HaJ3k5boPpNn+/5p9bWNps/oFmH0t68ICvt032A8hWve2D2pn9jdKfUuAI7Wj1h/n5+109bS6Xwa6eHcsmR9nc7YuhRCkC21yacmOL6xUXyib8ReQOEdkmIm+5vyt8ebeJyDoRWSMil5WrDR5JEx5+waMmpQFLhp9/dswd/wxff+wYjzqfQLXZkb06MgmM7WPV2O35YTNfo9v8gzX/ct4nYd9CziTMz982dhBlEpt9wDdae0q1XVzpjPAOtns3LpT7G77/Zoz5rj9BRE4FrgWmAKOAhSJyojGmvVyNcGb45jYnpPIl08/f7omQdGf4Smr7jjJRhGU+PuW2ZJt/u3/ZphHb6rHb/IPLlMrV00bYwHx2WeffbqfP1vijmH2s56TA/kYdu1AcbG/Xldqnav6V4WrgMWPMUWPMRmAdcFY5d+ho57lPsP/G92v+7QGxfYwT3yHLqwSgvkAzSX6unvZ92IJV2QZHPWzCPMvmX1tYf/KlNsJD2iPMUyPIzz+VH+iBFbyffImhPCmKbAWrcvuM4/hMuYX/F0RkuYg8KCLHuWmjga2+Mk1uWtnwNP9c+DU1f0hn2xe2HFdP1+ZPtmaZJizzOMr5zBUI9PMP0YhthyGSt08E76VSEOUhndkO+2SxbK8n22Sw7O1KafaJn0AphszDVUnNvzaGp6oos4+ILARGWLJuB+4F7sKRvXcB/wr8Ffa3YGNJQ0TmA/MBhg8fTiKRKKidbe3t7N//AS+9+EJouTffWMq+9bWYZJJW34BvW3uSrVu3kki8n0rbvfsILS1JtmzZQjLZTiKRoPnA4VR+65EjqeWlS5bwXt9oT4Dm5mZrP/cdSWalvf7aa2zs7dR7qLXjEL755puMqDvM1qVLO7b/4AMSiQTvbmnNqufoEafd7e3tqX1v23Y0rcyRI4dSy4sWLYrUl0J4Y+kS1u91+nqs9VjoOd+z2znG7733HonEB7S0tHDwYC0AmzZuZP9+x5LY5J67TRuPpbZdt3YtiSMbs+pc9X5bx4p7SN9du5bE4eyyuVi/2TnW/uNaalpaWspWd6Vpbj6ctr5q1Tv0+2BtVrlS9nnzZueaOHDgQJc+juU4z0UJf2PMxVHKiciPgKfc1SZgrC97DLA9oP77gfsBZs6caRobGwtq59dfXcDgQYNovGAW/H5BYLlZM2cydfQAav/wjPMlLzfUswHGjxtHY+PJqbK/2P4Ge9oPMGbMMOq2b6GxsZEfrHoF9n0AQP9+fXjvUAsAZ591FpOG9bXv9Jmn01YHDuhPY+OcrGI7m49A4g9paeecM5sxx/UGoOVoG/zhWQBmzpjBvvVvMfnkGfCy88AbMngQjY1nsfXVzfDOirR6+vfty46DzdTW1uId41cOrYJNG9LKbG9pBqDQ82Alo/9nnTWL9vV7YPU79KjvEbqvn297A97fwehRo2hsPI1EIkGfPjXQ3MwJJ5zAtvZd8MEexo93zt0q1sO7qwE4+eSTaDxrXFadx1a+B2+6D00BDJxy8kk0zsoum4stizfBqpVpx7XUJBKJstVdaf5t5cuwf19q/bQpU2g8bWRWuVL2eVnbu7B+LQMHDKCx8dyS1FkOynGey+nt4z9rHwM8ifMb4FoR6SkiE4DJwOvlagd0xOmJGisGMr/kZR+Mck3+AXb0wswkgTb/HDN8ixnwtdv809frKvRenM+Ar1j66VFbQ1qoh8xywR5Yucc/ohJDS0JJqWR4hzhSTm+fb4vIdBz5uAm4CcAYs1JEHgfeAdqAz5fT0wc6bP65riV/8DP/N3z9eR3rPldPS131BQ6QlmTA1+baWNPRt0xsgj3b1bMyvgF+L53cZaMO+GaXCzoltm4WanuO4yBiKanIDN8YS/+yCX9jzGdC8u4G7i7XvrP351xIOWPF+LTjDNmfrfnXdIR3sEn/QmfEBrqFWttr386mEXeErrC9peQe5LQNCpcDv39+LsIGfO1+/v5tgx6yOuDbVSinS7FHnE9RLKY2RPf2Cc6zBnZLEqj5p82IzcvbJ7pQSvuGr0Xzt78NhLc1qB2VuBG9/Ub29vG8eAIeXuJb9uru2DZ4/1HSIrUvxoKlFFTG2ye+Jykewt8SldN2zsOiSdrMPs7HwLNn/2aWz8vsE3BGcmn+9tg+FuFvqafOstPMPtVXzOZfOrNPdjl7fub+MylUPsRZsJSCShy+oLGfOBAP4U/2hRQWujeXoHXWvRm+uUMmVMLmn27SyE4TS5qH3eafvl45m3/+M3yD/PyzyqXNug4S/qXT/GMsV0pCJR6e3i6svuZVTjyEv22GbgRTR1j5VEjnALNPkCdOLgJj++SM55+9bAthHHWGb+axqJzNP7rQtA/kZvfTHtsneP9B+8kX1fyLQ80+5SUWwj+JxYYdpuFFePVPeftgN/ukaeJ5CM7AorY25fhgvH3ANxtPsIcJx65o87cJdeOO1NvevKK4epb0Yy6xuLtKSJaHXfl3GedxmXhcnpbAbGG+7bk0eW89M55/UPlymX0CBVgZYvtUSvOvEYksNPM1+/iPV6DZJ8KxiEqctcpSUBE//xhL/1gI/yTZwtP6eh9yIdgCu6Vi+0QUPlEINEdYy4bbrdNt/hJYj23At/O8faILzbABX1v45iimuFwPknxQP//iqMQlF3ZfVDuxEP62kMxhvu1WM07G5SHifszFOLmZFPrN23zi+QdV66XnCvbm4bXV+F67Mw+P7YMv5aBUfv62wfD0QfHox1ldPTuHSn7JK47EQ/gT1dsnW2P2yLb5iyv4g76+1ZGYj9ZcSs0/l5+/t9y1/Pyj35Bhg7vOpD6vTsvxyMfPv8C7JM7hgktBZWz+8T038RD+tm/wRohnk1Y+y+YfHt4hs2xU8nP1DK8jzfTknmn/G4y3FCm2Txcc8PVvE5bW4erZkR/s6pmdVrjNv6DN4kvGcdbYPuUlFsI/afLU/C112MM7dHwoJoySDPjmcPVMT3e3sbo7kpUWLbZPJYV/vtt0LHumqzQ3V8sAeD6B3TS2T+dQiQlYcT5HsRD+YNfcg8pE+ZSfiM/VM4fun8/1lcuOH62sRfO3be/+2wd809cr5udfk/8NGRTeIVWn5byW4g0rF/EVK6WhEiYZ7wGjk7yqFCe2T3paLoGRiS22j6mg5p/dntymC5umm/424ObZ/PwzDkZXnOHrYRfYFrNPjochRJvwFpU425NLQSUOX5znYsSi68nIfv55mH3E5+ppKe/fXT6vr1FnnobORnbzbB5H/q28NxabVt8dYvt0bJPdT9uErqAZ0UF1dZTNrz2pumJxd5UPneFbXmJxeeYd2yeHJumtpwZ8c1xA+VxfUW3+YQLS5tlj9WTyNP9IsX0qafPP0+xjNYllm3iiaP653EbzIc725FJQiYdnnM9RPIS/JfJmFAGfnpe+7sT2wYnnn4PS+PlHr9ML+2D187d4+0Rx9eyKsX08cg3SeothZq2Osmr26SpURvMv+y66LDER/tE02Y6LLbcZJDVQFGDzL/SaiurnH0Xzr7EIu3RvH+ffZs/PPl6VuVRqC7L5h7+55DfDN/dbUFTiLFhKgfr5l5d4CH/sZptMOjTE7DqCvIXak/ZJXoV6D0SdeRrlLaXGIugLtfl3xW/4ethKp7t6enWn78e+f0v9qvl3CurnX15iK/zz9erI0rzd7dtNblfPfIj6MZcoA742u7dd8899LCpl8xfJ/4a0PWg7ZmAHaf5Bwr90mr/K/jzJ8d3scqCxfaoc6ySvsNg+ljoyhbJXX3u7XfMv9GKKbvMPq6NjOfstIHtDu80/d5lykE9snzDsfv7+/GjHOaxs7jbEUaSUDjX7lJdYCH8nBENus0mtxS4eVN5bdzT/0hHs6hldE7dpuFbNP6SuzhrwdfadX3nrw9pi37cNAmeifv5dBx3wLS/xEP5EG/AN9aAJsPknk/aPuRRKvuGMc+VlerrYtrJF7Mx6U6pQVE8I9sTJB7/XULFmn0JPb5wFSymoSFRPL6Jt2ffU9YiH8DfZF1LYxCt7HJ3M9XDNv3CBEVX4R8vz+mKL9+PR3TV/ex02zd+XHzS2UkKzT5x9yAsiQMEqJ3F+O4uH8McyQzak51EEgHdjtyftYT1NnqpEmKeRvXz0txTwmX0s5btSSGcojdC0zvr1ewDlM+BbcEjnwrZTHNTsU17iIfwjhncIwxbeAZwQD6W4fupdCVMKzT+svK36sMHvVJkKakiluOn9VqrivX3U5t8ZVOLwxfkcxUP4E2y2sWE34wQM+AbY/PO9pjq+uxutfKEToWyb2Xz48xlgLjWl2JXtY/SFh3corA1xFiyloDKunmXfRZclFsI/GdHbxyPKFP+OAd/S+AjXpYR/8QO+YeVt4xlRZvhW8iYpxU2fM7xDoKtn7gdhVOIsWEqBBnYrL7EQ/lG9fcKwxfYBd8C3BNePF1wtqqDJ/80ieDvvwePP6szBylLsOmzQ21kOMvvY0lT4dwY64FteihL+IvJxEVkpIkkRmZmRd5uIrBORNSJymS99hoi87ebdI2WWMt6XnfIJj2Aj0M8/aZ/hm++s37oym31ss349rG6vedVeWoq5Ib0towZ7i7LvQsc74ixYSkElFJA4n6JiNf8VwJ8BL/gTReRU4FpgCjAX+KGI1LrZ9wLzgcnub26RbQglc4q/R5jbuu2CyPIW8g/4lkLzz9vsk1/9YX7+3oOnq/g6l0to2gLdZRLle8aF7E/Jnzh74lSCooS/MWaVMWaNJetq4DFjzFFjzEZgHXCWiIwE+htjFhtHJX8EmFdMG3KR9L7pmseAr42wAd9SUJfy9olWvtCY91G9fTqTYpoTdjb8fQ+eSW1pT4EN6mKHtdtRyYdnHE9VXZnqHQ286ltvctNa3eXMdCsiMh/nLYHhw4eTSCTybkibK5w3bdpIIrEtlb5n966ssl79R48cycp7e/lyzPba1PqaplYADjS3pLbdv/9wKn+Xr/4o7T521Nnn5s2bSSR25Cx/+PChwHoTiQQtLS0kEgmSySQA69etI9G6mZW721Pl2tud5XdWvA1AW1tbqs63d7al1fnOO6vy6k+hJBIJtrc4bW5tPRa6r23bjgKwbt06Em2baWlpoaXFOUdLly5h7x7nHC1/eznyXh1bDnT0/bXFixnYK1v3afM9zL23xtdeXcwgS9lceP3wH9dS453naqD5wOG09ZdfepEeFk+0Uvb53Q+ca2Lf/v1d+jiW4zznFP4ishAYYcm63RjzZNBmlrSgLx4GKmvGmPuB+wFmzpxpGhsbwxtr4UhrO/z+GSZOnEhj4yR45mkARgwfDu9tTyvr1d/w+vNw+FBa3hnTT2fOpCGp9b1vNMGKZTT07k1dTQ2Njefxg1WvwL4PABg6ZCi8/15avVbc9vTt05tdhw8yccIEGhsnh5Z1yvehsfECa35jYyOJRILGxkZqnlsA7UlOOnEyjeccT/263bDkNQDq6uqgvY0zz5gOf3qVurq6VFvb3nkf3liSqvrUU0+B5W/l7k+++Prk1b1hVwu8tIgePXqE7itxYCVs3sSkSZNo/NAEEokEffvWQvMBZsyYyfN71sDuXUw7bRqNJw9j9XsH4JUXAZgzZw5D+/XMqrM9aeD3v0tLm3PuuQzv3yvvrnn98B/XUuOd52rg31a+DPv3pdYvuOB8etbVZpUrZZ/7btoLry1mwIABNDaeW5I6y0E5znNO4W+MubiAepuAsb71McB2N32MJb1sBNv88zX7pK/7zT712ddn3kiIWcZG1Fdi70tjUWz+ofVUcECgFAO+frxjEO1jLpY61eZfGTohpHOcKZer52+Aa0Wkp4hMwBnYfd0YswNoFpHZrpfPdUDQ20NJ6Ljx09PD/fyz07LDOzj/7UlTkmBgHbNQI5bP21XVe7p0pHmLtRZXz86kfAO+ufehn3HsOujxKy/Funp+TESagHOAp0XkWQBjzErgceAd4Bng88YYz+D6OeABnEHg9cCCYtqQi2SA5h8mO21ZoSGdS3CNelVEdW8rOLyDNZ5/15ruUap7PrOaKJO8bKiff4XI4x5ViqeoAV9jzBPAEwF5dwN3W9KXAFOL2W8+JFN+/unpxcf2cRKCZvjme+N75cs+w7cbePuUS2im+fnn8bwrOLxDFzuu3Q2NilpeupbKVwaM43CRrbmH3JhRpvj7v+Frk1aF2siju3rmV6/F6pOiUt/njUpnmn1sFP4N34I2U5SKUPXCP1DzD7P5W9JCwzsU08Cs/USrLV+BlO8M386kEpO88pm1q4HdlGokNsK/+JDOds3f+ZJXdvlC7/vo3j751VsTEtunkuGao1CuZ5G/m/l0WW3+SjVS9cLfs77kFaUygrePt96W7ND8jc/Wc+2scZw3eQinjxkQqZ0dwcfKbPMPyctlqRrStwd/MXNsjlIloAihedMFEwEYO6h3Vp7/mNk+XRmEevtUL10lpElnUK4Zvl2GZEBgt3zNPll+/q7ssH3D9xc3n8PM4wdx/olD825v+QO7ZedFrWrJP1yS1z4LJd+geH6unj6aq6fbJ4373/Zyvfn17VlHy1FnlnPBb3EZ/4rSlah+zb9Mk7xsNv9LpzgToUcNbMi7nX16OjPFonqIFDqPwCaKGno4+553xqj8Ku1mRD1m/zN/Ns/dcn5qXTX/6mW0e69ecurwTm5J5YmN5p/lqpmnt09oSGc366bzJ/KXs8YxoHd93u0c2LtH4L5tRBUsre3p/bdt1qOuhuV3XEqfHt3rcvAeWj3qoukwniIwcUif0HJnTxyctq4DvpXh7y47iS8/9ia7W45VbJ+jBjaw7J8upX9D97r2S0HV99ib5JUdniF9fYQvdktKR5bgN4f0b/h2eNLkK/if+uKHeGfHARav35O271zY5mX9z/zZ7DmYfuMM69eTnc1HmXH8caH19++V/wOrlPzss2fTfLQtd0EfX7xoEvU1wicCxiFOGdmf59fsYmhf59wO69eT/3PpifzZmWOs5YMoWIir7M+LOZOGsOQfLmHl9v2s2La/YvstRFmrBqpe+Ad9zMWz+Z85biDzz5/ICUP7Zm1bI0J7wJuDJ/CTQeHqIjJ19ACmjh7AO9sPAG4gugjYBFKmxgrw2PzZ1NfWMKyfIwC72sSZz8wez6OvbuZcX9C8qPTuUcctl54UmH/LJSfy4VOGc5o76C4ifOGigKB5Fm66YCL/uWhD2T23lHSmjBrAlFHRHCWUwomB8Hf+gyZ51dXWMHfqyLS8jtm20J5KE2uZZIn8/Ae62seBw62RykcV4hMzHmr+rX58wyz+65VN9LWYe3q7YxDnnjCYGhEuLpNN9K55U7lrXvqE78F9enD51BH89YcmFFV3XW0NM8YfV/D2t11+CrddfkpRbVCUrkrVC/8gm3+4t49/QlSQ5u9g7BN882ZAgyP890UU/oXaof1tPWvCIM6aMMha7pyJg/nGn53GR08fRZ+elb1MamqEez89o6L7VJS4EQPh7/znE94hVUb8ywHSn+JcEz08zX/foWjCv9CJWVHbKiL85VnjCtqH4tDbjfX9d5ef3MktUZRsYiD87eEdooR0To//nmH28QnRUmj+49xJSbYPjNgo1HavdujKUVdbw6ZvXtnZzVAUK1Uv/AMHfCN4B6ZFgQyZIVwKgTpj/CAevGEm554QbeCzi4XjURSlmxED4e/8B37MJWR+t1+oZ5qJJG25NJL4opOjD6pqvBlFUYqh6mf4Btn8w2b4em8JYZ/9k5C3gkpQ6PdXdOKRoigQC+Ef7q1jw8vzb1Mq7b5UqM1fUZRiiI3wz/bTj+LtE6b551dXqSnY7NPFHmKKonQOVS/8PZt/psgLk52pr16lmXbCbP6VpxR+/oqixJeqF/5BH3MJNfv4Zvh6hGv+RTSwQArX/BVFUWIg/FPePpk9jTDDN8zP3y9GO0OgarwZRVGKoeqFf6DNP8K2aa6eAbF9bHVXgsK9dlT6K4oSC+Hv/Bdi80+b5JVxpNTmryhKd6bqhX9HYLbg8AyZeDnhmn8n+/mrzV9RlCKIgfBP58Y5xwPRBHaoq2fIWiUo3M9fxb+iKDEU/v/8kSls+uaV4eI6NcO3Iync5l+69kWlYLNPaZuhKEo3JXbC3yPke+a+Gb4hgd062dsn3w/Qe6jirygKFCn8ReTjIrJSRJIiMtOXfryIHBaRt9zffb68GSLytoisE5F7pJPsEFFmuqZp95mGnk7X/HWGr6IohVOs5r8C+DPgBUveemPMdPd3sy/9XmA+MNn9zS2yDYWRp7dPmKLdGQJV/fwVRSmGooS/MWaVMWZN1PIiMhLob4xZbJxA+48A84ppQ6FEC+wWPMmru2r+iqIoUN54/hNE5E3gAPAPxpgXgdFAk69Mk5tmRUTm47wlMHz4cBKJRN6NePcD5xPsy5cvI7m9o7trtjmfS9y/b19WvQcOHAbg0KGDqbRFixJpnjJbDrSnlnft2lVQ24phW9NWEomdgfktLS3WNu0+nEwtV7rN5Saoz9WM9jkelKPPOYW/iCwERliybjfGPBmw2Q5gnDFmj4jMAH4tIlOwK9yBn1MxxtwP3A8wc+ZM09jYmKu5WfTdtBdeW8zpp5/OeZOHptL3LG2Ct5cxYOBAGhvPSdvmB6tegX0f0K9vX2g+gAhceOGFaWVW7TgAr7wIwPBhw2hsPDPvtuXNM0+nFsePG0dj4ymBRROJBLbjtW3fYVj0RwBrfncmqM/VjPY5HpSjzzmFvzHm4nwrNcYcBY66y0tFZD1wIo6mP8ZXdAywPd/6S0HoDF/334sHZDOxpCV1ggWmYD//ErdDUZTuSVlcPUVkqIjUussTcQZ2NxhjdgDNIjLb9fK5Dgh6eygr+Uzysg32drarp4Z3UBSlGIp19fyYiDQB5wBPi8izbtb5wHIRWQb8ArjZGLPXzfsc8ACwDlgPLCimDYUSGt4hI56/TcvuroHd1NVTURQocsDXGPME8IQl/ZfALwO2WQJMLWa/pSDc7OMKfXfdrvnblyuFav6KohRDbGf4RsETsLls/p0hUNXmryhKMVS98A9yJQoVnhlCP5eJpXM0fw3uoyhK4VS98PfICs8QWtahJmXzDyvVWTb/wrZTm7+iKBAj4Z9JFHmdGvjNsX2nzPAtUPoX+tBQFKW6iK/wj+DtkzL7WCRm+oBvN/L20RFfRVGIs/CP4O0TPskrONxzJdB4/oqiFEN8hX+EMuGTvPKrq9QUrvmXuCGKonRL4iv8I4R0jj7Jq5Qti0bhzj4q/RVFiYHwN4Fh46LY/NP/07cW63KlUFdPRVGKoeqFv0fWZxjziu3T9TR/neGrKEoxxEb4ZxLu559u6885yasbuXqq7FcUBeIs/D2JbTELZdv8g8u4ayVtWxQKDu+gqr+iKMRY+EdRnMNj+6irp6Io3ZfYCv/i4/nblyuFunoqilIM8RX+od4+6QO9XdHVU2P7KIpSDFUv/E2Qr2c+sX26oKtn4Tb/EjdEUZRuSdULf49MmZdPVM+u6eqpUlxRlMKJjfDPJExzjjbJy75cKdTPX1GUYoiv8I9QJvRjLmmaf/eZ4as2f0VRIM7C35OBIXI9NLZPJwvRgid5qexXFIU4C/9I3j6k/aeXsS9XCvXzVxSlGOIr/IuM7ZNWVzcK7KYzfBVFgTgL/whlOj7mEr69av6KonQ3ql74FxDROTU3IDyev9/Pv/Kon7+iKMVQ9cI/RWZI5wgiu8PfP7y67uTnr2YfRVEgTsI/g9LG8+8Mm3/Fd6koShURW+Efhmcq8gRs7vAOlUdn+CqKUgxFCX8R+Y6IrBaR5SLyhIgM9OXdJiLrRGSNiFzmS58hIm+7efdIF7ZDhNn8O3uKb9c9aoqidAeK1fyfA6YaY6YBa4HbAETkVOBaYAowF/ihiNS629wLzAcmu7+5RbahbISGdPabfbqRq6eiKAoUKfyNMb83xrS5q68CY9zlq4HHjDFHjTEbgXXAWSIyEuhvjFlsHJeaR4B5xbShHHiBQEM/5uJf7kYDvoqiKAB1Jazrr4D/cZdH4zwMPJrctFZ3OTPdiojMx3lLYPjw4SQSibwbtWpPOwDL3lrGsa21qfTVe530/fv2ZdW7d+8Rp3FNTW6ZD7LKHG7rcCLdumULicR7ebetGJYvX0bbttrA/JaWlpzHq5Dj2ZWJ0udqQ/scD8rR55zCX0QWAiMsWbcbY550y9wOtAE/9TazlDch6VaMMfcD9wPMnDnTNDY25mpuFj3X74E/vcr06dM554TBqfReG/bA668yYOBAGhvPSdvmoQ2vw+5djB8/FjZtYNCgQTQ2np1W5uDRNlj4LADjx4+jsfHkvNuWN888nVo8I6M/mSQSCQKPl1tPIcezKxPa5ypF+xwPytHnnMLfGHNxWL6IXA9cBXzYdHw5pQkY6ys2Btjupo+xpHcpOrx9Irp6dorNv+K7VBSliijW22cu8PfAR40xh3xZvwGuFZGeIjIBZ2D3dWPMDqBZRGa7Xj7XAU8W04ZyEj7Jy+fq2QmCuFalv6IoRVCszf/fgZ7Ac6475KvGmJuNMStF5HHgHRxz0OeNMe3uNp8DHgYagAXur0uR95e8KtCm7P2r8FcUpXCKEv7GmEkheXcDd1vSlwBTi9lvucme5JVD0OoMX0VRuhk6wzeEzLj+6Xm+5Qq1x4+6eiqKUgxVL/xNsDNRTjwBmzO8g/r5K4rSzah64e+Rj6z0nJZCJ3l1srePyn5FUYohNsK/ELzv5OoMX0VRqg0V/iGIpP+n53VyVE89c4qiFIGKkBBCXT39y6r5K4rSzVDhH0KHzT+8nH7MRVGU7oYK/xCiTvLqDHSSl6IoxVD9wr9wT8/oH3BXs4+iKN2M6hf+LoGi0vJwyI7nn6tuNfsoitK9iI3wL4Qws4+fSinhn5k9PrVcjObft2cd150zPndBRVGqllJ+zKV7EiJDU5p/jkdkpbTwu+ZN5aYLJnLfovWMHNCr4HpW3HlZ7kKKolQ1KvwteCEhQj/g7qOS9vcxx/Xm6/NOq9j+FEWpTtTsE0Iqtk+Ocup5oyhKd0OFfwhhsX1s5RRFUboLKvwteN4+EtnbR1EUpXtR9cK/CDf/lAtnTpu/qv6KonQzql74exRklxdH+8/t6qnCX1GU7kVshH8+pMw+7i+XYq+Kv6Io3Y3YCv/+veoBmDC4T2AZEUFEcpp1OmOGr6IoSjHE1s//1FH9eeiGWZxzwuDAMp7mn8uqo5q/oijdjdgKf4ALTx5mTe+Y5OXY+3O7eqr0VxSlexFr4Z8LEfj6vKmcOf64nOUURVG6E1Uv/E0xIZ0RPjFrTO5yKv0VRelmxGbANx/5nDnJKxdq81cUpbsRG+FfTtTmryhKd0OFfwhRzTkq+xVF6W4UJfxF5DsislpElovIEyIy0E0/XkQOi8hb7u8+3zYzRORtEVknIvdIFzSYe8MEURvWBbugKIoSSrGa/3PAVGPMNGAtcJsvb70xZrr7u9mXfi8wH5js/uYW2YayoTZ/RVGqlaKEvzHm98aYNnf1VSDUNUZERgL9jTGLjTEGeASYV0wbyknUmbtq81cUpbtRSpv/XwELfOsTRORNEVkkIue5aaOBJl+ZJjetbJhC4nqqt4+iKFVOTj9/EVkIjLBk3W6MedItczvQBvzUzdsBjDPG7BGRGcCvRWQKdjN6oHQWkfk4JiKGDx9OIpHI1dwsVuxuB+CtN9/k4KbaSNvs238YgJUrV9J7z5qc5VeuXEmv3bnLVZKWlpaCjld3RvscD7TPpSGn8DfGXByWLyLXA1cBH3ZNORhjjgJH3eWlIrIeOBFH0/ebhsYA20P2fT9wP8DMmTNNY2NjruZmUfvuLljyOmeccQYzjx8UaZsfrl4MH+zltKlTaJw6MrjgM08DcNrUqTROsT0fO49EIkEhx6s7o32OB9rn0lCst89c4O+BjxpjDvnSh4pIrbs8EWdgd4MxZgfQLCKzXS+f64Ani2lDV0Bt/oqidDeKDe/w70BP4DnX3fFV17PnfOBfRKQNaAduNsbsdbf5HPAw0IAzRrAgs9KuQ8QBX50toShKN6Mo4W+MmRSQ/kvglwF5S4Cpxey3UkRV6DWev6Io3Q3VWS2kQjpHLK9WH0VRuhtVL/yLiuoZUaqrzV9RlO5G1Qt/jwK/3x4JFf6KonQ3YiP88yHfkM4q+xVF6W6o8A9Bhb+iKNWKCv8QNLaPoijVigp/C6kx4sixfVT4K4rSvVDhH4K6eiqKUq1U/QfciyG6q2eZG6IoCq2trTQ1NTFgwABWrVrV2c2pKFH63KtXL8aMGUN9fX2kOqte+Bfi5u/Gp9MveSlKF6KpqYl+/foxePBg+vfv39nNqSjNzc3069cvMN8Yw549e2hqamLChAmR6oyR2Sd/AR09nr8Kf0UpN0eOHGHw4MGqbFkQEQYPHsyRI0cibxMj4Z8/Ub199FJUlMqggj+YfI+NCn8LqQ+4q+avKEqVosI/BPX2URTFo7a2lunTpzNlyhROP/10vve975FMJsu6z2PHjvGVr3yFadOmMWnSJK666iq2bNlSkrqrfsC3KFTzVxTFpaGhgbfeeguAnTt38slPfpL9+/dz5513lm2fX/va12hububNN99k4MCBPPTQQ1x99dUsXbqUmiI/JKLC30Iqtk9Um7/KfkWpKHf+diXvbD9Q0jpPHdWff/7IlEhlhw0bxv3338+sWbO44447OP/88/nBD37A9OnTAZgzZw733nsvv/rVr9iyZQsbNmxgy5YtfOUrX+FLX/oSAPPmzWPr1q0cOXKEL3/5y8yfPz9tH4cOHeKhhx5i48aNKXv+jTfeyIMPPsjChQu59NJLi+pv1Zt9TBExndXmryhKEBMnTiSZTLJz504++9nP8vDDDwOwdu1ajh49yrRp0wBYvXo1zz77LK+//jp33nknra2tADz44IMsXbqUJUuWcM8997Bnz560+tetW8e4ceOy3FpnzpzJO++8U3T7Y6P5lzekc/51K4pSOFE19HLjKZcf//jHueuuu/jOd77Dgw8+yA033JAqc+WVV9KzZ0969uzJsGHDeP/99xkzZgz33HMPTzzxBABbt27l3XffZfDgwWl12zx4ilFo/cRG+OdDh7dPVLOPSn9FiRsbNmygtraWYcOGISJccsklPPnkkzz++OMsWbIkVa5nz56p5draWtra2kgkEixcuJDFixfTu3dvGhsbs3z0J02axObNm2lubk5Lf+ONN7jmmmuKbn/Vm32KQUM6K4piY9euXdx888184QtfSCl/n/3sZ/nSl77ErFmzGDRoUOj2+/fv57jjjqN3796sXr2aV199NatMnz59uP7667nllltob28H4JFHHqFXr17MmTOn6D6o5h+CfslLURSPw4cPM336dFpbW6mrq+Mzn/kMt9xySyp/xowZ9O/fnxtvvDFnXXPnzuW+++5j2rRpnHTSScyePdta7hvf+AZf/epXOfPMMzl69ChDhw5l8eLFJbE2qPC34cX2iTzgW8a2KIrSJfC07yC2b99OMplM88K544470sqsWLEitbxgwYKc++zZsyf33HMPd999NwcPHmTu3Lk8+uijWZ5BhaDCPxT9mIuiKLl55JFHuP322/ne975XtP99ECNGjEjNMygFVS/8ixkXV5muKEoUrrvuOq677rrObkZexGbANx85nvL2iVi+Ru0+ilIRSuXmWI3ke2xiI/wLQT/moihdh169erFnzx59AFjw4vn36tUr8jZVb/YpBvX2UZSuw5gxY2hqamLfvn15Cblq4MiRIzn77H3JKyoq/C3kq1io6FeU8lNfX8+ECRNIJBKcccYZnd2cilKOPhdl9hGRu0RkuYi8JSK/F5FRvrzbRGSdiKwRkct86TNE5G037x7pwtNjo0/y6rJdUBRFsVKszf87xphpxpjpwFPAPwGIyKnAtcAUYC7wQxGpdbe5F5gPTHZ/c4tsQ6ejNn9FUbobRQl/Y4w/pmofOhxlrgYeM8YcNcZsBNYBZ4nISKC/MWaxcUZtHgHmFdOGchLV/KOav6Io3Y2ibf4icjdwHbAfuNBNHg34g1U0uWmt7nJmelDd83HeEgBaRGRNgc0ccua32J3vRtO/Fa3c4IjlKswQyL/P3RztczzQPufHeFtiTuEvIguBEZas240xTxpjbgduF5HbgC8A/4x9DNSEpFsxxtwP3J+rjbkQkSXGmJnF1tOd0D7HA+1zPChHn3MKf2PMxRHr+hnwNI7wbwLG+vLGANvd9DGWdEVRFKWCFOvtM9m3+lFgtbv8G+BaEekpIhNwBnZfN8bsAJpFZLbr5XMd8GQxbVAURVHyp1ib/zdF5CQgCWwGbgYwxqwUkceBd4A24PPGGC8k3ueAh4EGYIH7KzdFm466IdrneKB9jgcl77PoVGlFUZT4obF9FEVRYogKf0VRlBhS1cJfROa64SXWicitnd2eUiEiD4rIThFZ4UsbJCLPici77v9xvjxrqI3uhIiMFZHnRWSViKwUkS+76VXbbxHpJSKvi8gyt893uulV22cPEakVkTdF5Cl3var7LCKb3LA3b4nIEjetvH02xlTlD6gF1gMTgR7AMuDUzm5Xifp2PnAmsMKX9m3gVnf5VuBb7vKpbt97AhPcY1Lb2X0ooM8jgTPd5X7AWrdvVdtvnHkxfd3leuA1YHY199nX91tw3Mefcterus/AJmBIRlpZ+1zNmv9ZwDpjzAZjzDHgMZywE90eY8wLwN6M5KuB/3KX/4uOsBnWUBuVaGcpMcbsMMa84S43A6twZodXbb+NQ4u7Wu/+DFXcZwARGQNcCTzgS67qPgdQ1j5Xs/AfDWz1rYeGkqgChhtnHgXu/zA3veqOg4gcD5yBowlXdb9d88dbwE7gOWNM1fcZ+D7wdzgu5B7V3mcD/F5ElrphbaDMfa7meP55hZKoYqrqOIhIX+CXwFeMMQdCgupVRb+NMz9muogMBJ4Qkakhxbt9n0XkKmCnMWapiDRG2cSS1q367DLHGLNdRIYBz4nI6pCyJelzNWv+QSEmqpX33aipuP873fSqOQ4iUo8j+H9qjPmVm1z1/QYwxuwDEjgh0Ku5z3OAj4rIJhxT7UUi8hOqu88YY7a7/zuBJ3DMOGXtczUL/z8Bk0Vkgoj0wPm+wG86uU3l5DfA9e7y9XSEzbCG2uiE9hWFGw7kx8AqY8z3fFlV228RGepq/IhIA3AxTgiVqu2zMeY2Y8wYY8zxOPfsH40xn6aK+ywifUSkn7cMXAqsoNx97uxR7jKPoF+B4xWyHicKaae3qUT9+m9gBx0hsv8aGAz8AXjX/R/kK3+7ewzWAJd3dvsL7POHcF5tlwNvub8rqrnfwDTgTbfPK4B/ctOrts8Z/W+kw9unavuM45G4zP2t9GRVufus4R0URVFiSDWbfRRFUZQAVPgriqLEEBX+iqIoMUSFv6IoSgxR4a8oihJDVPgriqLEEBX+iqIoMeT/A2AbZFy1+nQaAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot rewards\n",
    "plot_rewards(\"Cliff World\",rewards, 'Dyna Q')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " v  >  >  >  v  v  >  >  v  >  v  v \n",
      " v  >  v  >  >  v  >  v  >  v  >  v \n",
      " >  >  >  >  >  >  >  >  >  >  >  v \n",
      " X  C  C  C  C  C  C  C  C  C  C  T \n"
     ]
    }
   ],
   "source": [
    "# print policy \n",
    "print_policy(env, agent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dyna Q Learning for \"Taxi\" environment "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAEICAYAAAC3Y/QeAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAx7ElEQVR4nO3deZwU9Z3/8ddneoaZ4VJADmWQQ/EAlqiMRjzHlSAxJhqNWXMsmsQQTWIOd5OI7v6C67q5do1h3cQlRllzGRMlJjEmkcSWHBij0aCCHCrgCAiiHCMwMMz390dVz1TXVHdXHzPdM/N+8ujHVH3r+n57mE9961vf+pY55xARkf6lqtwZEBGRnqfgLyLSDyn4i4j0Qwr+IiL9kIK/iEg/pOAvItIPKfhLr2NmD5nZ5eXOR19jZovN7N/LnQ/pGQr+0q3M7Dkza/E/B81sX2D++kL26Zx7u3Pu/2IeP2lmzszeEkr/qZ/eVEgeJB4zuz3w+24xs1Yz251lfWdmbwbWv6Mn89ufVJc7A9K3OeempqbNLAl8zznX03/Qa4C5wD/5+RgBnAps6+F8dGFm1c65tjIcN+GcO9jdx3HOXQVcFTjuYqA9x2Zvcc6t6858iWr+/YKZHWFm95nZNjN7ycw+FVi2wMzuNbO7zWy3X1Nv9JddZ2Y/Ce3rG2a2sAR5OsrMfmdm283sNTP7vpkdGlj2upmdFMj/a6laul+bvzKPw30f+AczS/jz7wOWAPsD+anyy/uCn6d7zWx4YPmPzWyLme00s2VmFjypLTaz/zGzB/3v8M9mdlSGck/wa7cfMbONwO/89A+b2Soze8PMfm1m4/30G83sv/3pGr9W/FV/vt6/khoWM4/fMrNfmtmbwDlmdqKZ/dXP84+Aujy+07yZ2SDgEiDWVZt0LwX/Ps7MqoCfA38DxgLnAp8xs/MCq70LuAc4FPgZcJuf/kPgfDMb6u8rAbwX+IE//00z25HhsyJX1oAvAUcAxwPjgAUAzrkXgC8A3zezgcBdwGLnXLLAr2ETsBKY7c/PBe4OrfMp4CLgbD9PbwD/E1j+EDAZGAX8Fe+EEvQ+4EZgGLAOuDlHns7GK/d5ZnYRcD1wMTAS+D3edw/wKNDkT58MbPG3BZgJrHbOvREzj+/38zUEeBz4KfBdYDjwY7zAHMnMzsjyu95hZmfkKC/+/rcBy3Kst8w/id1vZhNi7FcK4ZzTpw9/gLcCG0Np84G7/OkFwNLAsinA3sD8H4C5/vTbgBeKyEsSuDLDsouAp0JpPwOeAVYAtXH2k+mYwAfxAuqxwBp/WTPQ5E+vAs4NbHc4cACojtjnoYADDvHnFwN3BJafDzyfIT8T/G0nBdIeAj4SmK8C9gDjgXpgHzACuA7vJNEMDMY72SzMcJyoPN4dWH4W3knRAml/Av69G/8v/hZYkGOds4ABfv5vA56N+h3oU/xHNf++bzxwRLCWhhdARgfW2RKY3gPUmVnqftAP8Gq14NUcf1CKTJnZKDO7x8xeMbNdwPeAw0KrfRuYBvy3c661yEPeD/w9cA1ebTdsPLAk8B2tAg4Co80sYWZf9puEdgHr/W2C+Q1/h4Nz5Ofl0LG/ETj263hXRmOdc3uBJ/Bq+2fhXQn8CTjdT3sUvKuyGHkMHvMI4BXnR1zfhhx5LpiZjfPzG77iSuOcW+ac2++c2wF8GpiId4UkJabg3/e9DLzknDs08BninDs/5vY/BprMrAF4N4Hgb117cgQ/z+XY75fwaqbTnXND8WrmFtj3YOBW4DvAgmD7eyGcc3vwathXEx38XwbeHvqe6pxzr+Cd9C4EZgGH4NXeCea3kCyFjv2x0LHrnXN/8pc/infiOhH4iz9/HnAKnU0ocfIYPOZmYKyZBZcfmSmzZnZmlt91i5mdmaO8c4E/OedezLFemKO471kyUPDv+x4HdpnZF/wbhAkzm2ZmJ8fZ2Dm3Da/p5C68k8iqwLKrnHODM3ymZtqnbwjQAuwws7HA50LLvwE86Zy7EngQuD1qJ4EbqBNiFOd64Gzn3PqIZbcDNwdutI40swsDeW0FtgMDgf+Icax83A7MT92gNbNDzOzSwPJH8YLnSufcfjqbsl7yfz+F5HE50AZ8ysyqzexivJNJJOfc77P8rgc7536f43hz8ZqeMjKzqWZ2gv9/dDDwX8AreFdhUmIK/n2c87rzvRM4AXgJeA24A692GNcP8GqUJWny8d0InATsxAvu96cW+EF3Dp1dBK8FTjKzD0TsZxxec8UruQ7onNvknPtDhsXfwLvH8Bvz+qE/hne/BLymitQxVvrLSsY5twT4CnCP32TzLPD2wCp/wmv7T9XyV+LdBwjeOM0rj/5J5GLgCryb2/9A4HdQSmY2E2jAu4oML3vIOp/3GA38CNgFvIh39XKBc+5Ad+Srv7P0Jj+R3sXM/gXY5pz733LnRaQ3UfAXEemHytbsY2ZzzGy1ma0zs+vKlQ8Rkf6oLDV//2GhNXj9xpvxejC8zzm3ssczIyLSD5Wr5n8KsM4596J/4+kevG5qIiLSA8o1sNtY0h84aaazZ0UHM5sHzAOor6+fMW7cuIIO1t7eTlVVac5z63d1jknVMLiK5pZ2BlYbowYau/c7tu/zrqSM9E7VAMPrjKEDjNf3OXbtT19aZdDuJ9X6I9C0xhx2a8gA79gAw2qNN1od4wZBIlHFrv2O1/d1Hiuch5oqGDOoipd3d5ZrzMAqtuzx5usS3v637fXWHTu4Ku17mDC083vdfcCxfa9LS2/e3U6b8/ZpBjVVXlkzSeVrWK1hwOutjqEDjOF1ubt6B3/Pbe3Q3NKeVmbwvuONu7vmPa6Djo7vKmr7vW1Qn+Wvamer441Wx6AaY2R98d3XS/F/2wEbIn6flaj1IGx+s50xg6qoS+Rev68o5ve8Zs2a15xzI8Pp5Qr+Uf/ru7Q/OecWAYsAGhsb3RNPPFHQwZLJJE1NTQVtGzbhugc7pu+5aiaX3r6co0YO4rf/1MT3HtvAv/z02YzbLnjnFK44fSJf+MkKHlm9lQumH8Gdf3wJgCF11YwaUssL296kcfwwntjwRsb9hM1/+3F86aHnAXjpS+fT1u744++X0dTUxA/+vJHrlzzTse5/XvoW3jOjgVseXsPC365l6hFD+fknz+CaHz7Fg89sBuAX15zBBf/t9Yg859iRfOj0icy983GOHT2EX3/2rLTvYfWX39Gx79VbdnPercvS0t/6H0t5dVcrv/rMmRw3ZmjOsnwzuY6v/mo1N797GvsOtHPTL1ZyxWkTWPCuXI8NpP+ed+45wFv+7TcA/Nelb+GSGQ2AN5zJxPm/7JL3uJxzfPZHT3Np4zhOPzr8QHJuP3x8I/Pvf4YLph/Obe8/Ke/tw0r1f/v6Jc9wzrGjeNuU0blXLrOlv3uEWX9/Trmz0aOK+T2bWeST2+UK/s14/bNTGvDGGak4ydVbOW7MUMYc0nXAwzfe9AaFHFJXk9c+DxxsZ0B1FYNrE2lps44fzYQRLTy/JXq48+DVQdBJ44d1TJsZNYnOc2t1qJo9uNb7lQ8a0Hnsqirjfz5wEg/6Af3QgZ3lGVhbzeC6an/f2ct17JghLP7QyVQFVmw76GX40PoB2Tf2feSMiQxIVPHexnH8+cXXAZg2Np9HEjypPIenUw+0jj20Pu99pra/9bITC9oWvJM8wMGoX2QZ/ce7/67cWYgt/H9aClOu4P8XYLKZTcR7KOUyvMfTK84Vd/2FUUNqefyGWV2W7djrPXuS+oOO8+ccvMFeWxMM/o6aRBW1NVW8smNv5LZHHFpP8xudy44dPYT7P35axvUBaqrT/1BSeR3onwSi7vcfOrAzUA8akGBAwrvcDAb148YM4bgxQ7ps23TsqLT5a2cfww1LnmXYoHgnyNrqBFeeOQmAMyYfxsOfPYujR+UaJqerRCBADKlL/2/+2PxzqR9QnjaDVOBqq7DgL/1PWYK/c67NzD4J/BpIAHc653KNBdNj1m1toWFYPXV+cN66O3pMsZ17vOA/tN4PbDl6Tv3y2S0s+PlKphw+FDOore5swzvY7gX/uurMQWnM0Lq04D9+xEAG1VYzqDbzr/Ht0w7nzy++zj1/8W6xDPSDXqrm3x7I883vnsbwgQMYNCBBdZXR1u4YOKC6I1AeFQjCv/rMWVnLmvKBt47nA28dH2vdKJNHdz3BxDVoQII39x/suNpJibqK6ykjBtcC0DCssCsPkVIp25u8nHO/BH5ZruNnsu/AQWbd8iizjh/NHZc3Zl33jT1es8/Qunhf43Ov7ARg5eZdHDl8YMfJJaU6YWlXA+FmntFDo4PW4AGZj19Xk+DLl0zn6FGD+fcHVzFyiBd8BkZsEwzSh9TXsP3N/QyqTXDUyMF8e24jpx01InchK8hnZh3Dzb9cxWF+wK0EJ08Yzu0fPKnLFVJPOXDgAM3Nzezbt68sxy+FQw45hFWr+tdwP3HKXFdXR0NDAzU18a6y9RrHkNTl+PIXXotcXpMwDvjt2G+kav5+m3+uC/nBddW8ud/rwmMGA6rT794PSFR1XA1UGZx21GH8YV1nPobW1zD1iKGMHFJLcnXnGwgH1uZuwrjyzElcfFIDwwd5TTqD/G0yXazUD0jAm50nid5wIzDso2dN4pIZnWWuFHOmHV62Yzc3NzNkyBAmTJjQcf+jt9m9ezdDhhR+Rdgb5Sqzc47t27fT3NzMxIkTY+2zsvt1lUGuh95qA80yO/yafypA5npeLliLN7p2eapJWMfVwMghtZw3NT3g1tVU8eCnzuQfGseFtov3awwGwY48ZzhlpU5Cg8rUNl4qlRb4y23fvn2MGDGi1wZ+iWZmjBgxIq8rOgX/PFUHetLs2JPfYIPBE0vUH19NdWfNf8zQOj546nju/dhMzj7G66KbOjGU4g+3vibV5h+9/Ay/G+OufT3+bnHpZgr8fVO+v1c1++SpPRAtU719UrXnXFcNuTp41CSqOgL8qKF1mBmnTBzOt5Le8mw3gz921iSmNxya/QABqZNYpjxfO/tYtuzaxwXTy9dEISLdRzX/DDKdRYPNCG+2erXiVPzM1eYf7FkTtXev2aez5p/i32LoWBZl/vnH8448AvUwvzvn2cdE33g8pL6G//3HRiaNzL+bpUg2iUSCE044galTp/KWt7yFW265hfb29twbFmH//v185jOf4aijjuLoo4/mggsuYOPGjZHrTpgwgddei77n1x3OP/98duzY0WPHS1HNP0+jhtaxfvseAPb4N2/j9thOq2RbehMSeDX/1D2FYHfE1NVGuHdQMUYOqeUPXzgn7SQj0hPq6+t5+umnAdi6dSvvf//72blzJzfeeGO3HfP6669n9+7drFmzhkQiwV133cWFF17Ik08+WbKhXzJpa2ujujpzqP3lL8vT6VE1/3wFAvie/X57uHPBHxmFa/7n/116Td1r9vF+JcFunW1+rSi1LHX1MX7EwLyzH9QwbCDVMW8Wi3SHUaNGsWjRIm677Tacc5x55pkdJwaA008/nRUrVrBgwQI+/OEP09TUxPTp01m4cGHHOhdddBEzZsxg6tSpLFq0qMsx9uzZw1133cXXv/51EgmvAvWhD32IwYMHs3Tp0lj53LZtG5dccgknn3wyJ598Mn/84x8BePzxxznttNM48cQTOe2001i9ejUAixcv5tJLL+Wd73wns2fPZvHixVx88cXMmTOHyZMn8/nPf75j36krjfXr13P88cfz0Y9+lKlTpzJ79mz27vWe63nyySeZPn06M2fO5HOf+xzTpk3L74uOoJp/EcI1/1xXAMGTg5lRW51gwTunsODn3kjWAxJVHPRr98EaeeqKOFXzP2XicL5zeSNnTu4yVpNIbDf+/DlWbtpV0n1OOWIoX3xn7nGYgiZNmkR7eztbt27lyiuvZPHixdx6662sWbOG1tZWpk+fzv3338/zzz/PI488wubNm5kxYwZXX301NTU13HnnnQwfPpy9e/dy8sknc8kllzBiROczKevWrePII49k6ND0saUaGxtZuXIls2fPzpnHT3/603z2s5/ljDPOYOPGjZx33nmsWrWK4447jmXLllFdXc3SpUu5/vrrue+++wBYvnw5K1asYPjw4SxevJinn36ap556itraWo499liuueYawoNVrl27lh/+8Id8+9vf5r3vfS/33XcfH/zgB/n4xz/OHXfcwWmnncZ115Xm9ScK/nmK6hrZHvOdCFFt/lWBYQhqElUcPWowp0wczrSxnf9RD/rbBbuZnnt87+t3L5JJquPBpZdeyk033cTXvvY17rzzTq644oqOdd7xjndQW1vLiBEjGDVqFK+++ioNDQ0sXLiQJUuWAPDyyy+zdu3atODvnIu8h5fPu0yWLl3KypWdrxvZtWsXu3fvZufOnVx++eWsXbsWM+PAgc4egG9729sYPnx4x/y5557LIYd441RNmTKFDRs2dAn+EydO5IQTTgBgxowZrF+/nh07dtDS0sJpp50GwPvf/35+8YtfxM57Jgr+eYr6/9Jxwzdnb5+uy4P/KWsSxrjhA7n3YzPT1mnraPNXE42UTr419O7y4osvkkgkGDVqFGbG2972Nh544AHuvfdegiP51tZ2PqmdSCRoa2sjmUyydOlSli9fzsCBA2lqaurS1/3oo49mw4YNXR6U+utf/8p73vOeWHlsb29n+fLl1NenD8txzTXXcM4557BkyRLWr1+fNvLmoEGD0taNyn9YeJ29e/fmdZLKh6JJAcKjCsb91aQ95OXvIrirTO3vqV9+KW/4ilSCbdu2cdVVV/HJT36yoyJ05ZVX8qlPfYqTTz45reYcZefOnQwbNoyBAwfy/PPP89hjj3VZZ9CgQVx++eVce+21HDzoNdXefffd1NXVcfrpp8fK5+zZs7nttts65lP3JXbu3MnYsWMBr52/OwwbNozBgwd3lO2ee+4pyX4V/PPkSB8xEnLf6O1cL9js4+0jOFLmgAzB/2A39PYRKZe9e/d2dPWcNWsWs2fP5otf/GLH8hkzZjB06FA+9KEP5dzXnDlzaGtrY/r06fzrv/4rp556auR6X/rSl6ivr+fYY49l7Nix3HLLLTzwwAMZu3RPnz6dhoYGGhoauPbaa1m4cCFPPPEE06dPZ8qUKdx+++0AfP7zn2f+/PmcfvrpHSeW7nDbbbcxb948Zs6ciXOuo/moGGr2CQnG8ajLLecc1VVGcJzPzoe8su87V80/PPxySir4Zzo5iPQmuYLkpk2baG9vT7sRu2DBgrR1nn2286VJDz30UM5j1tbWsnDhQhYuXMiWLVuYM2cO3/3ud5k3b16XddevXx+5jx/96Edd0mbOnMmaNWs65m+66SYArrjiirT7FeH5YJt96niHHXZYWrn++Z//uWP6+OOPZ8WKFQB8+ctfprEx+6CTcSj4F6Aq/DKJmDX/XG3+1Rn6G6eCf/iKQ6Svufvuu7nhhhu45ZZbuq3//ZgxY9K6k/YGv/71r7n11ltpa2tj/PjxJWliUvDPIvLmLpnb/DMNkpZtf1WhG75RUr19FPylr5s7dy5z584tdzYqziWXXJJ25VAKakfIIFOYdQ4SoRqJi/mQV9r+LdXm35lWlaH98T3++2dTY/GLFKO7eo9IeeX7e1XNP4tMX2W46b2Qv6WOfv6BgN+lOcl39dlHceUZk7qM/y+Sr7q6OrZv365hnfuY1Hj+dXXxh2tR8M+T1+xT1SUt+DOO1N9d8O8vkeGP0cwYkOFmsEg+GhoaaG5uZtu2bblXrlD79u3LK8j1BXHKnHqTV1wK/iHBWnymy6hw23t7Qc0+qZ/Bmn/87UUKUVNTE/tNT5UqmUxy4oknljsbPao7yqxwky+/q2coKW+d/fw70zLV/EVESk3BP8xFTqalZep1k6u3T5Rgm79684hIT1HwzyRLHO76hG/hzT5pvX0U/EWkhyj4Z+IyD+LWJfgXsPvUHoJt/mr2EZGeouAfkvNBLUrT5k9HP381+4hIz1Pwz8QynwjCzTNxX+Ae2r23rxgPeYmIlJqCf0iu+O1cxPAOhfT26WjzV81fRHqegn+esrX5F3MSgPSrABGR7tRtwd/MFpjZK2b2tP85P7BsvpmtM7PVZnZed+WhEOlDOkevE3zCt66mqqCxUqKGd9Dj9iLSU7r7Cd+vO+f+M5hgZlOAy4CpwBHAUjM7xjnXfW9CKKFwP/+6mkTnaxzz2I9F3PAVEekp5Wj2uRC4xznX6px7CVgHnFKGfESKU4tPBX8zqK2uCrzDN/5xom74ioj0lO4O/p80sxVmdqeZDfPTxgIvB9Zp9tMqSuYhnV1Hbb22ugrDCnqyN2psHxGRnlJUs4+ZLQXGRCy6AfgWcBNea8hNwH8BHyY6rkZGTzObB8wDGD16NMlksqB8trS0xN52V6uXlba2Nh5dtqwjPbV9S8seXm/fgwEJ2tm/v5UfP9HMWLeNl3a2x87Tzp07SSaTrH69s7Wr0PJFyafMfYXK3D+ozKVRVPB3zs2Ks56ZfRtIvbSyGRgXWNwAbMqw/0XAIoDGxkbX1NRUUD6TySRxt32tpRUeWUp1dTVnnXkWPPwrgI7tBz+9jJEjBpJ4bSuD6gZQXWW4ffv4+pOtfPrcybBubazjHHrIoTQ1zWTQ+tfh8eVpxyiFfMrcV6jM/YPKXBrd2dvn8MDsu4HUm4l/BlxmZrVmNhGYDDzeXfnIV9qQzhEXJKnlVVVGXU2i8GabiLF9RER6Snf29vmqmZ2A16SzHvgYgHPuOTO7F1gJtAGf6C09fcA7IRhGdZVRW13Fnv0HA8viixrbR0Skp3Rb8HfO/WOWZTcDN3fXsYsRrO1n6r1j5g3CVleTYO+BwHkrn+EdIp7wFRHpKXrCN0/BZp/a6qouwTtuLI96mYuISE9R8M8i08tczLy+/l6bf/b1c1HNX0TKQcE/LEYEN4wqM7+ff2BTl/UdMOn7iHiBu4hIT1HwzyLqad9UWqKKyN4+cW/gqs1fRMpJwT8kVtONQW11gvoBifSafx4NP6k2f8V+ESkHBf8MzCxjmz/AV98znY83HZXWzlNIs49q/iJSDt09qmevk7O3ph/gT500Auga7PON5ertIyLloJp/AYLt+sHpwl7lq+gvIj1PwT8k10Ne4aQuNf+YDT8az19EyknBP0/OubTwbqE2/7g0nr+IlJOCf0haAM8yvENKVVqzT/w7vrrhKyLlpOCfQaaYnKtyH7u3T47jiIh0JwX/kPSKf/SQzunNPoWN76A2fxEpJwX/AqT19gmkp8b9yYeCv4iUg4J/SHBIh+jePumJ4dgdu7eP/1M3fEWkHBT8C5C5t0/+4/mrn7+IlIOCfxaRwzuEEoM1fefyafbReP4iUj4K/iG5Ku/OkVb179rsE4+6eopIOSn4Z5GpGSdY2w/f8M2Xgr+IlIOCfwY79hzgJ08251wvbWwfl8d4/qmf+g2ISBko9GTxpYee75LmnEtr6lGzj4j0Rgr+IXE67FiG6UJe5qIbviJSDgr+eeoyqmeo2Udj+4hIb6DgH5Kr9h7uztl1SOd49AJ3ESknBf8CpPX2KTJ4q+YvIuWg4B+Ss59/eHiHUF0/fm8fDewmIuWj4F+AjMM9F/A2F93wFZFyUPAPyfn+9tAKaa99JH4zUOd4/or+ItLzigr+ZnapmT1nZu1m1hhaNt/M1pnZajM7L5A+w8ye8ZcttF4W/XIF+Pg3fHtVsUWkjym25v8scDGwLJhoZlOAy4CpwBzgm2aW8Bd/C5gHTPY/c4rMQ0nFa7qJDtyFvMNXRKQcigr+zrlVzrnVEYsuBO5xzrU6514C1gGnmNnhwFDn3HLnRdm7gYuKyUNPyxbgHU41ehHpFaq7ab9jgccC881+2gF/Opweyczm4V0lMHr0aJLJZEGZaWlpib3tljfbI9NT2+/f38rmzZtIJrcDsGPH3o51Nr2yiQMH2mIdZ+vWV9PyNOmQqoLLFyWfMvcVKnP/oDKXRs7gb2ZLgTERi25wzj2QabOItEzPv2asSzvnFgGLABobG11TU1P2zGaQTCaJu+0L21rg9492SU9tP+APDzP2iDE0Nf0dALet+hPseAOAw484ggGvb4H9+3MeZ8zo0TQ1nejlb9qbHDaklsG1pTsX51PmvkJl7h9U5tLIGW2cc7MK2G8zMC4w3wBs8tMbItJ7ja69fdKnY/f2Caw44bBBRedLRCQf3dXV82fAZWZWa2YT8W7sPu6c2wzsNrNT/V4+c4FMVw9lkfshr1wBPr8hnUVEyqHYrp7vNrNmYCbwoJn9GsA59xxwL7AS+BXwCefcQX+zq4E78G4CvwA8VEweyiH91Y3ZX/guIlKJimpkds4tAZZkWHYzcHNE+hPAtGKO271yDeyWbbmLP9aPqv4iUkZ6wjdP4WafLkM8x9xPeEwgEZGepOAfku/LXPLdtmMfiv0iUkYK/nnK+pCXy39sHxGRclDwD4k1uEOWCB+3OUc1fxEpJwX/PIVv+AZn83mHr4hIOSn4h8Tp559N/GYfVf1FpHwU/AuQqbdP+MTx0KfP5PtXvjXnPkREepqCf76yjuqZfiO3riZBXU0icl0FfxEpJwX/kFzt9l6AD1b9Q+/0DUR1I1uQV/QXkfJR8C9A5nf4ps9XWeaWfdX8RaScFPxDct7wDff2SZsOXwV03V4vbBeRSqDgn6dMLyVICQf88DMBVf68zgEiUk4K/iGxhnfIOL5D1/XCq3YEf0V/ESkjBf88ZX+Hb3pQrzLrEuSr/G9c/fxFpJwU/EPiPKUbbMoJnwyCQd2r+Wdo9lHsF5EyUvDPU7aTQ/hmcFTtPqE2fxGpAAr+Ibl7+6QH7uDJINzsYxZ1A7joLIqIFE3BvxBZAnhwUWRXz6pUs4/OAiJSPgr+eQpfGKSN6hnR/t/lhq+CvohUAAX/fLnMPXW8Zp/0G75huuErIpVAwT8k337+XWv76dNde/uklin6i0j5KPjnKZ/ePlH9/BMa30FEKoCCf57CvX26yNHbR80+IlIJFPxD4j3klWnb0Hr+v6htFftFpJwU/PPUpbdPaCYtqOuGr4hUKAX/kFg3fLPU24O9faoimn0S6ucvIhVAwT9PXcbzd8EnfLu+1Ssc4tXsIyKVoKjgb2aXmtlzZtZuZo2B9AlmttfMnvY/tweWzTCzZ8xsnZkttAqrAueq+IeHcEhb5iK6eoZr/pVVXBHpp4qt+T8LXAwsi1j2gnPuBP9zVSD9W8A8YLL/mVNkHnpc3Je5eNPRo3qq6i8i5VRU8HfOrXLOrY67vpkdDgx1zi13XnvJ3cBFxeSh1MLNOl2Xx18WNZRDZ+xX9BeR8qnuxn1PNLOngF3Avzjnfg+MBZoD6zT7aZHMbB7eVQKjR48mmUwWlJGWlpbY276w42BkenD7DRs2kExu9ve9pyN922vb2PNme8f8smXL2L43/Yyw5803AXh540aSyS2x8lSIfMrcV6jM/YPKXBo5g7+ZLQXGRCy6wTn3QIbNNgNHOue2m9kM4KdmNpXoxo6MdWnn3CJgEUBjY6NramrKld1IyWSSuNsO3fgGPPanLukd2//qQSZMmEBT0zEADH56GbTsBuCwww7jTdvTMX/22Wfxyht74Q+PduznkKFDYPdOxo8/kqam4woqTxz5lLmvUJn7B5W5NHIGf+fcrHx36pxrBVr96SfN7AXgGLyafkNg1QZgU777L5eoJqHco3pGj+0jIlJO3dLV08xGmlnCn56Ed2P3RefcZmC3mZ3q9/KZC2S6eiiLOG36cZ/wrYp6gbuiv4hUgGK7er7bzJqBmcCDZvZrf9FZwAoz+xvwE+Aq59zr/rKrgTuAdcALwEPF5KEcgjdro/r2R02naDx/EakERd3wdc4tAZZEpN8H3JdhmyeAacUct3tlGbUz15YRwzurn7+IVCI94VuAzOP5uy6vccw0sJuISDkp+Och1zMAEH7IS69xFJHKpOAfkvWGr//TItJybZtSpW9cRCqAQlEBslXec1XsVfMXkUqg4B+SrfKeq2YftTgc6ytsHDsR6acU/POQ6taZKYA75yJu8IbmuydrIiJ5UfAPidNun75+5wbtLqKmH1pfFX8RqQQK/nkoRbOP2vxFpBIo+Ifk250zvbePy9mso9EdRKQSKPgXINNY/O2ua7tPxIscuylXIiLxKfiH5N3bJ8uonhDV7FNIrkRESkvBvwCZmu3bI5p9dMNXRCqRgn9I9id8I8bzD0ynevtc9/bjaBhW7yV26f2j6C8i5dedr3HsczrG88+4gvfjqrOP4qqzj4pcRcM7iEglUCgKiardh+XX7JP9oS8RkXJQ8M9DrtNCu3Ndn+jN8dCXiEg5KPjnIfUMQNqbvAI3CSIf8grN6yEvEakECv5hMYZ3yNzsE9W7Ry9zEZHKo+Cfhxzd/L0nfDWks4j0Agr+IUUN6Rz1kFeOeRGRclDwL0CwKScY8Nsjh3TOvK2ISLko+Idkrd0XVPNXm7+IVB4F/wKkv8M3OJ5/1B3f9FmN7SMilUDBPyTbQ165HgCLiv1hGt5BRCqBgn8eOoZ3yBC/o04OXUb1rEqtKyJSPgr+IXFe45ip7n6wvWtXz1z9/kVEykHBPw9R54WjRg7uXO5yj+VjoZ8iIuVQVPA3s6+Z2fNmtsLMlpjZoYFl881snZmtNrPzAukzzOwZf9lCq7CqcPZ+/v7wDoEsL3zficyeMjrjthrPX0QqUbE1/4eBac656cAaYD6AmU0BLgOmAnOAb5pZwt/mW8A8YLL/mVNkHnpcMIAPravhrGNGAqmB3bJvqyd8RaQSFBX8nXO/cc61+bOPAQ3+9IXAPc65VufcS8A64BQzOxwY6pxb7rxq9N3ARcXkodSyvcA905JUPI8K/hrVU0QqUSlf5vJh4Ef+9Fi8k0FKs592wJ8Op0cys3l4VwmMHj2aZDJZUMZaWlpib7tiW1tkejKZZGerF/7Xrl1LsnV9x7K1Lx8AYO/efbzxRmvasVoPpp8yNm16BYANGzaSTG6JWYL85VPmvkJl7h9U5tLIGfzNbCkwJmLRDc65B/x1bgDagO+nNotY32VJj+ScWwQsAmhsbHRNTU25shspmUwSe9vVW+HJv3RJbmpqYtvuVnhkKccccwxNp47vWLb58Y3w3DPU1tYxfNggmpre2rFs34GD8PCvOubHNYyDDS9x5PgjaWo6rqDyxJFXmfsIlbl/UJlLI2fwd87NyrbczC4HLgDOdZ1tJs3AuMBqDcAmP70hIr1iZB/dIXpp6owWp81fTf4iUgmK7e0zB/gC8C7n3J7Aop8Bl5lZrZlNxLux+7hzbjOw28xO9Xv5zAUeKCYPPSrDO3xTAT2fZwR0DhCRciq2zf82oBZ42O/++Jhz7irn3HNmdi+wEq856BPOuYP+NlcDi4F64CH/UzkKeJlLqm9/e0T07/qEr8K+iJRfUcHfOXd0lmU3AzdHpD8BTCvmuOWS8bzQ0dsn6qGu6Ie8RETKSU/4huQavA2yBfTcNf/UyUFj+4hIOSn45yFTm34qoEe9wzdMrT4iUgkU/EOy3bRNXRVkenAr6gExDe8gIpVIwb8AmQK61+YfXhZ9D0DnABEpJwX/kKw1/4zNPt5P7x2+oWWheTX7iEglUPDPQyr2Z+rqGXUXVzd8RaQSKfgHrNu6my279uVcL9NL2b0nfLNX7TWqp4hUglIO7NbrzbplWdbl2Ub8hOjePl3a/BX7RaQCqOafh47Yn7EpJ84zAiIi5afgX4BMN3Wjevt02dai9yEi0pMU/Eugc2C3TKNWB9dV2BeR8lPwL0CmvvtxRvVMUW8fESknBf88xOrnr4q9iPQCCv556BjeIZSe1ubfozkSESmMgn8Bcr2kXUSk0in45yFzm35n9I97ItD5QkTKScE/DxmHdwjMh5/+FRGpRAr+BSjF27nU20dEyknBPw+ZhncIdv1U+7+I9AYK/nnIPKqniEjvouBfAmlt/joTiEgvoOCfh1wPeYFu+IpI76Dg78s1XHNQpuEdRER6CwV/X3us2J+p6p9hWkSkQin4+w7GiP6pi4Ns7+lV7BeR3kDB39eeV7NPeF4hX0R6FwV/X5zgn2mNtJq/TgQi0gsUFfzN7Gtm9ryZrTCzJWZ2qJ8+wcz2mtnT/uf2wDYzzOwZM1tnZgutQqJlnDb/zmafzO/lrYjCiIjkUGzN/2FgmnNuOrAGmB9Y9oJz7gT/c1Ug/VvAPGCy/5lTZB5KIk6bf0rXh7wU8kWkdykq+DvnfuOca/NnHwMasq1vZocDQ51zy53Xt/Ju4KJi8lAqcbp6ZnpBux7yEpHeppRt/h8GHgrMTzSzp8zsUTM7008bCzQH1mn208pOvX1EpD+pzrWCmS0FxkQsusE594C/zg1AG/B9f9lm4Ejn3HYzmwH81MymEh0bM4+SbzYPr4mI0aNHk0wmc2U3UktLS85td7ZmD/7JZJKXd7cD8Nxzz1G/fXXHslXbD3ZMv/rqq1mP9dKLLwKwccNGksktOXJeuDhl7mtU5v5BZS6NnMHfOTcr23Izuxy4ADjXb8rBOdcKtPrTT5rZC8AxeDX9YNNQA7Apy7EXAYsAGhsbXVNTU67sRkomk+Ta9tVd++CR32Zc3tTUxMpNu+CPv2fatKk0TTu8Y1ntC9vhL48BMGbMGJqaTkjf+FcPdkxOnDQJ1q7myPFH0tR0XN5liStOmfsalbl/UJlLo9jePnOALwDvcs7tCaSPNLOEPz0J78bui865zcBuMzvV7+UzF3igmDyUSj43fMMXMOrtIyK9Tc6afw63AbXAw36Pzcf8nj1nAf9mZm3AQeAq59zr/jZXA4uBerx7BA+Fd1oO8fr5Z7jhW+rMiIh0s6KCv3Pu6Azp9wH3ZVj2BDCtmON2h/b23Ot03PDN9oSvzgQi0gvoCV9fXsM7hOcV8EWkl1Hw9x3MI/iHpXf11JlARCqfgr8v1kNeHc0+WW74KvaLSC+g4O87GKPNPyUc37ft3h9/W50cRKQCFNvbp88oprfPWccc1jEdFdv/5R3H0/zGXiaPHsxFJ4xl4/Y9XN10VKFZFREpmoK/L6/hHUIRfuCAaj5yxkS+84eXImv2V545KW3+y5dMLzSbIiIloWYfXz73e6MCfHWV2nNEpPdQ8PfF6e2TbY0qP/irt4+I9AYK/r78+vl3DfAJ/3JAN3RFpDdQ8Pe1x2rzz7xOlZp9RKQXUfD3xXqNY2oiS5t/PlcQIiLlouDvy+s1jhFpCT/45/O8gIhIuSj4+/J5wjdKld/YH2c/IiLlpuDvize2j7dOeHgHgIT/TarZR0R6AwV/Xz7vcolu9vG+yoOK/SLSCyj4++L19sm8LGHx9yMiUm4K/r54Y/t4ovryd97wVfAXkcqn4O/Lr7dP1+hfpa6eItKLKPj7YvXzz7KO+vmLSG+i4O/La3iHiGafVFdPNfuISG+g4O+L1eafZZ1ER82/ZFkSEek2Cv6+WOP5+z+zPeGrZh8R6Q0U/H15xewsvX0U/EWkN1Dw9+XzJq8oCbX5i0gvouDvy+cdvtm7epY2XyIi3UHB31dsb59UzV9P+IpIb6Dg74sVs7M1+/jjO8QbIE5EpLwU/H1F9/YxNfuISO9RVPA3s5vMbIWZPW1mvzGzIwLL5pvZOjNbbWbnBdJnmNkz/rKFFjU+chnkMw5/9JDOavYRkd6j2Jr/15xz051zJwC/AP4fgJlNAS4DpgJzgG+aWcLf5lvAPGCy/5lTZB5KotjePlWmrp4i0nsUFfydc7sCs4PobBm5ELjHOdfqnHsJWAecYmaHA0Odc8udV9W+G7iomDyUSl7j+Ue9wzehrp4i0ntUF7sDM7sZmAvsBM7xk8cCjwVWa/bTDvjT4fRM+56Hd5UA0GJmqwvM5mHAawVu6+XlK53Tp3wl83obAPtsMUcqmaLL3AupzP2Dypyf8VGJOYO/mS0FxkQsusE594Bz7gbgBjObD3wS+CLR90RdlvRIzrlFwKJceczFzJ5wzjUWu5/eRGXuH1Tm/qE7ypwz+DvnZsXc1w+AB/GCfzMwLrCsAdjkpzdEpIuISA8qtrfP5MDsu4Dn/emfAZeZWa2ZTcS7sfu4c24zsNvMTvV7+cwFHigmDyIikr9i2/y/bGbHAu14zd1XATjnnjOze4GVQBvwCefcQX+bq4HFQD3wkP/pbkU3HfVCKnP/oDL3DyUvs+XTv11ERPoGPeErItIPKfiLiPRDfTr4m9kcf3iJdWZ2XbnzUypmdqeZbTWzZwNpw83sYTNb6/8cFlgWOdRGb2Jm48zsETNbZWbPmdmn/fQ+W24zqzOzx83sb36Zb/TT+2yZU8wsYWZPmdkv/Pk+XWYzW+8Pe/O0mT3hp3VvmZ1zffIDJIAXgEnAAOBvwJRy56tEZTsLOAl4NpD2VeA6f/o64Cv+9BS/7LXARP87SZS7DAWU+XDgJH96CLDGL1ufLTfeczGD/eka4M/AqX25zIGyX4vXffwX/nyfLjOwHjgslNatZe7LNf9TgHXOuRedc/uBe/CGnej1nHPLgNdDyRcC/+dP/x+dw2ZEDrXRE/ksJefcZufcX/3p3cAqvKfD+2y5nafFn63xP44+XGYAM2sA3gHcEUju02XOoFvL3JeD/1jg5cB81qEk+oDRznuOAv/nKD+9z30PZjYBOBGvJtyny+03fzwNbAUeds71+TIDtwKfx+tCntLXy+yA35jZk/6wNtDNZS56bJ8KltdQEn1Yn/oezGwwcB/wGefcriwjgveJcjvv+ZgTzOxQYImZTcuyeq8vs5ldAGx1zj1pZk1xNolI61Vl9p3unNtkZqOAh83s+SzrlqTMfbnmn2mIib7qVX/UVPyfW/30PvM9mFkNXuD/vnPufj+5z5cbwDm3A0jiDYHel8t8OvAuM1uP11T792b2Pfp2mXHObfJ/bgWW4DXjdGuZ+3Lw/wsw2cwmmtkAvPcL/KzMeepOPwMu96cvp3PYjMihNsqQv6L4w4F8B1jlnLslsKjPltvMRvo1fsysHpiFN4RKny2zc26+c67BOTcB72/2d865D9KHy2xmg8xsSGoamA08S3eXudx3ubv5Dvr5eL1CXsAbhbTseSpRuX4IbKZziOyPACOA3wJr/Z/DA+vf4H8Hq4G3lzv/BZb5DLxL2xXA0/7n/L5cbmA68JRf5meB/+en99kyh8rfRGdvnz5bZrweiX/zP8+lYlV3l1nDO4iI9EN9udlHREQyUPAXEemHFPxFRPohBX8RkX5IwV9EpB9S8BcR6YcU/EVE+qH/DxqFcVRHzBVkAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# create taxi environment\n",
    "env = gym.make(\"Taxi-v3\")\n",
    "\n",
    "# create a Q Learning agent\n",
    "agent = DynaQAgent(alpha=0.25, epsilon=0.2, gamma=0.99, get_possible_actions=lambda s : range(env.nA), n=50)\n",
    "\n",
    "#train agent and get rewards for episodes\n",
    "rewards = train_agent(env, agent, episode_cnt = 500)\n",
    "\n",
    "#plot reward graph\n",
    "plot_rewards(\"Taxi\", rewards, 'Dyna Q Learning')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion\n",
    "\n",
    "We see that Dyna Q Agent learns the optimal policy faster as compared to Q-learning. As the learnt policy does not have any exploration, the agent learns the shortest route of walking across the maze, right on the row just above cliff. As the actions are deterministic and there is no exploration, the agent has no chance of falling into cliff when it takes a RIGHT action in a cell next to the cliff. Therefore, it learns to take the shortest route towards goal.\n",
    "\n",
    "For Taxi environment with n-50, we can see that Dyna Q learns the optimal policy by 1-- episodes which is much faster than what we saw in vanilla Q-Learning. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
